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FOREWORD 


One  of  the  primary  concerns  of  the  Battlefield  Information  Systems  Technical  Area  of  the 
Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  is  with  intelligence 
systems-improved  tactical  intelligence  information  processing  to  aid  command  decisions.  The 
effective  integration  and  utilization  of  fragments  of  intelligence  data  from  many  sources  requires 
that  the  intelligence  analyst  evaluate  the  quality  of  available  data  appropriately  and  communicate 
the  evaluation  accurately  to  others.  The  research  reported  in  the  present  Technical  Paper 
investigated  analysts'  use  of  a variety  of  rating  scales,  including  the  standard  6-point  Reliability 
and  Accuracy  scales,  and  explored  the  basic  judgmental  dimensions  on  which  analysts  appeared  to 
evaluate  the  quality  of  data.  Previous  work  has  been  reported  in  ARI  Technical  Paper  260, 
"Subjective  Interpretation  of  Reliability  and  Accuracy  Scales  for  Evaluating  Military  Intelligence," 
and  ARI  Research  Memorandum  74-14,  "An  Inductive  Taxonomy  of  Combat  Intelligence  Data." 


Research  in  this  area  is  conducted  as  an  in-house  effort  augmented  by  conracts  with 
organizations  selected  for  their  unique  capabilities  and  facilities  for  research  in  intelligence 
systems.  The  present  research  was  conducted  jointly  by  personnel  of  ARI  and  Syracuse  University 
Research  Corporation,  Syracuse,  New  York,  under  contract  DAHC  I9-73-C-00I8.  Work  was  done  in 
response  to  requirements  of  Army  Project  20062I0IA754, "Intelligence  Information  Processing," 
with  the  cooperation  of  the  U.  S.  Army  Intelligence  Center  and  School,  Fort  Huachuca,  AZ. 
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BRIEF 


Requirement: 


To  explore  and  establish  the  concepts  that  underlie  an  intelligence  analyst's  judgments  of 
intelligence  information  quality,  and  to  develop  rating  scales  that  permit  accurate  and  uniformly 
understandable  ratings  of  information  quality. 

Combat  intelligence  is  seldom  derived  in  a simple,  straightforward  manner  from  information 
available  to  an  intelligence  staff;  it  must  be  produced  through  careful  analysis  of  many 
fragmentary  and  frequently  contradictory  pieces  of  data.  An  indispensable  aid  for  such  analysis  is 
a knowledge  of  the  quality  of  the  data.  Either  the  source  of  the  data,  or  someone  close  to  the 
source,  must  evaluate  the  data  to  enable  the  analyst  to  focus  on  the  implications  of  high  quality 
data  and  discount  low  quality  data.  The  evaluation  of  the  quality  of  intelligence  data  is  currently 
reported  on  two  6-point  scales:  a source  reliability  scale  and  an  information  accuracy  scale,  as 
defined  in  Field  Manual  30-5,  "Combat  Intelligence."  Evidence  suggests  that  these  scales  are  not 
an  effective  vehicle  for  assignment  or  communication  of  quality  ratings.  As  a consequence,  the 
ratings  are  frequently  misused  or  omitted,  and  information  of  potential  value  to  the  analyst-a 
realistic  rating  of  information  quality  be  someone  close  to  the  data  source-has  been  lost. 


Procedure: 

Two  groups  of  Army  Intelligence  personnel  evaluated  a set  of  intelligence  reports,  using  50 
different  ratings  scales  that  included  the  current  6-point  Reliability  and  Accuracy  scales.  One 
group  of  21  enlisted  personnel  had  just  completed  the  U.  S.  Army  Intelligence  School  course  for 
intelligence  analysts  (96B),  and  the  other  group  of  34  enlisted  personnel  were  just  starting  the 
same  course.  Most  of  the  reports  were  from  the  files  of  the  28th  Infantry  Division  for  10-15 
December  1944  before  the  German  Ardennes  counteroffensive  began  16  December  1944. 
Participants'  responses  were  analyzed  to  determine  the  characteristics  of  the  judgments  made  in 
evaluating  the  reports  and  the  relationships  among  the  scales,  and  to  derive  a taxonomy  of  reports 
based  on  the  rating  scale  interrelations. 


Findings: 

Analysts  did  not  successfully  discriminate  between  information  Accuracy  and  information 
Reliability;  both  ratings  are  used  to  represent  some  idea  of  Accuracy.  Furthermore,  this  underlying 
concept  of  information  Accuracy  may  be  better  represented  by  the  0-100  scale  of  Global  Validity 
than  by  either  of  the  two  standard  scales. 


Analysis  of  the  participants'  ratings  shows  three  basic  factors  or  dimensions  in  the 
psychological  space  which  determined  the  way  these  intelligence  messages  were  evaluated;  that  is, 
the  analysts  (both  trained  and  untrained)  judged  each  message  on  three  independent  factors.  The 
primary  factor  deals  with  the  ACCURACY  or  truthfulness  of  the  message.  In  evaluating  this  aspect 
of  message  quality,  the  analysts  made  use  of  the  standard  Accuracy  and  Reliability  scales,  the 
Global  validity  scales,  and  bipolar  scales  such  as  True/False,  Probable/Improbable,  and 
Acceptable/Unacceptable.  The  ratings  which  defined  the  second  factor  suggest  that  the  analysts 
held  a concept  of  information  RELEVANCE  or  importance.  In  evaluating  this  aspect  of  message 
quality,  they  used  bipolar  scales  such  as  Heavy/Light,  Many/Few,  Large-scale/Small-scale,  and 
Risky/Routine.  The  third  factor  in  message  quality  was  tentatively  identified  as  DIRECTNESS; 
here  the  analysts  used  bipolar  scales  such  as  Interpreted/Uninterpreted,  Implied/Unimplied,  and 
Understandable/Confusing. 


Utilization  of  Findings; 

The  findings  can  be  used  to  guide  the  development  of  scales  which  will  more  explicitly 
communicate  analyst's  evaluation  of  the  quality  of  intelligence.  Several  modifications  in 
procedures  are  suggested  by  the  obtained  Accuracy/Relevance/Directness  structure  of  ratings.  In 
particular,  unambiguous  scales  to  evaluate  these  aspects  of  information  quality  may  be  developed 
to  replace  the  current  6-point  scales  of  Accuracy  and  Reliaoility.  The  present  findings  and  the 
usefulness  of  any  new  scales  should  be  validated  with  a sample  of  more  experienced  intelligence 
personnel  and  with  a different  scenario. 
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INTRODUCTION 

Improvements  in  the  technology  and  techniques  of  intelligence  data 
collection  have  imposed  additional  burdens  on  both  the  intelligence 
analysts  and  the  decisionmaker.  Analysts  must  deal  with  the  increased 
volume  of  information.  The  decisionmaker  is  increasingly  remote  from 
raw  unprocessed  information  and  therefore  is  forced  to  rely  on  others  for 
evaluating  the  quality  of  data.  To  assist  users  of  intelligence  data  in 
discriminating  the  valuable  from  the  worthless,  a system  for  evaluating 
the  quality  of  Intelligence  data  was  developed  during  World  War  II. 

The  system,  described  in  DA  Field  Manual  30-5,  Combat  Intelligence, 
requires  an  examination  of  incoming  information  for  pertinence  and  time- 
liness and  provides  for  explicit  ratings  of  source  reliability  and 
information  accuracy. 

Considerable  anecdotal  evidence  and  some  experimental  evidence  show 
these  procedures  as  inadequate.  Present  research  examines  the  underlying 
conceptual  structure  characterizing  analysts'  ratings  of  tactical 
intelligence  data.  The  research  also  examines  the  role  of  the  current 
Reliability  and  Accuracy  ratings.  An  alternative  approach  to  rating  the 
quality  of  intelligence  data  is  suggested. 


Current  Ratings  of  Information  Quality 

Upon  receipt  of  an  intelligence  report,  the  analyst  must  first 
determine  whether  the  information  contained  in  the  message  or  report  is 
pertinent  to  his  unit,  and  whether  it  is  timely  or  has  been  overtaken  by 
events.  The  analyst  must  also  evaluate  the  reputation  of  the  source  or 
reporting  agency  for  submitting  factual  reports  (reliability)  as  well  as 
the  factual  nature  (accuracy)  of  the  data  being  processed. 

Evaluations  of  pertinence  and  timeliness  presumably  are  reflected 
in  the  analyst's  decision  to  continue  an  examination  of  the  data. 
Accuracy  and  reliability  evaluations  result  in  explicit  judgments  on 
two  standard  rating  scales.  These  ratings  are  attached  to  the  data; 
other  users  of  the  data  determine  the  quality  and  usefulness  of  the 
data,  at  least  in  part,  on  the  basis  of  these  ratings. 
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Reliability  is  judged  on  the  basis  of  a 6-letter  system  and  Accuracy 
is  judged  on  the  basis  of  a 6-number  system,  as  follows: 


Reliability  of  Source  and/or  Agency 

A * Completely  reliable 
B = Usually  reliable 
C » Fairly  reliable 
D = Not  usually  reliable 
E * Unreliable 

F **  Reliability  cannot  be  judged 


Accuracy  of  Information 

1 = Confirmed  by  other  sources 

2 - Probably  true 

3 ” Possibly  true 

A ” Doubtfully  true 

5 = Improbable 

6 - Truth  cannot  be  judged 


There  are  several  problems  with  the  present  system  of  evaluation. 
Analysts  seldom  agree  on  the  precise  meaning  of  the  descriptive  words 
and  terms.  Possibly  because  of  the  confusion,  ratings  are  frequently 
omitted  (Baker,  McKendry,  and  Mace,  1968).* 

In  addition,  although  analysts  are  specifically  Instructed  to  rate 
reliability  and  accuracy  as  separate  items,  individual  analysts 
consistently  use  either  reliability  or  accuracy  as  the  basic  criterion 
of  message  quality  and  correlate  the  other  ratings  with  it  (Meeland  and 
Rhyne,  1967;  Samet,  1975).** 


As  a result,  ratings  tend  to  be  A-l,  B-2,  C-3,  etc.  Furthermore, 
most  analysts  use  the  Accuracy  scale  as  their  basic  criterion  and  may 
tend  to  Inflate  their  ratings  or  not  report  items  of  low  accuracy/ 
reliability.  Nearly  three-fourths  of  all  reports  are  rated  B-2  (Baker 
et  al.,  1968;  Samet,  1975). 


Research  Approach 

The  approach  taken  to  uncover  the  psychological  bases  for  quality 
judgments  was  to  ask  a number  of  trained  and  untrained  intelligence 
analysts  to  make  many  ratings  of  quality  of  each  of  AO  intelligence 
messages.  These  ratings  were  examined  for  patterns  that  would 


*Baker,  J.  D.,  McKendry,  J.  M.  and  Mace,  D.  J.  Certitude  Judgments  in 
an  Operational  Environment.  ARI  Technical  Research  Note  200. 

November  1968.  (AD  681  232) 

**Meeland,  T.,  and  Rhyne,  R.  F.  A Confidence  Scale  for  Intelligence 
Reports:  An  Application  of  Magnitude  Estimation  Scaling.  Menlo  Park, 
Calif.:  Stanford  Research  Institute  Technical  Note  A923-31. 

June  1967. 


2 


Samet,  M.  G.  Subjective  Interpretation  of  Reliability  and  Accuracy 
Scales  for  Evaluating  Military  Intelligence.  ARI  Technical  Paper 
260.  January  1975.  (AD  A003  260) 
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indicate  the  underlying  criteria  being  used  by  the  analysts.  An  initial 
conceptual  analysis  of  the  information  evaluation  task  identified  six 
types  of  judgments  that  might  be  made:  These  judgments  concerned  (1) 
data  source,  (2)  message  content,  (3)  situation,  (4)  action  reported 
considering  the  overall  tactical  situation,  (5)  inferences  drawn  from 
the  data,  and  (6)  characteristics  of  the  message  itself.  Forty-six 
bipolar  scales  (e.g.,  Direct/Indirect;  Widespread/Local;  Garb led/Clear) 
were  generated  to  represent  these  general  concepts;  these  46  scales,  the 
standard  Reliability  and  Accuracy  scales,  and  two  0-100  "Global 
validity"  scales  were  used  by  the  analysts  tc  evaluate  the  40  messages. 

The  message  set  consisted  of  33  messages  from  the  files  of  the  28th 
Infantry  Division  for  the  period  10-15  December  1944  (just  before  the 
German  Ardennes  Counteroffensive  of  16  December  1944)  and  an  additional 
7 fictitious  and  misleading  messages  generated  for  the  experiment.  The 
raters  were  21  "trained"  enlisted  personnel  who  had  just  completed  the 
U.S.  Army  Intelligence  Center  and  School  (USAICS)  course  for  intelli- 
gence analysts  (96B)  and  34  "untrained"  enlisted  personnel  who  were  just 
entering  the  same  course.  Most  of  the  participating  personnel  had  had 
little  Army  experience  beyond  Basic  Training  and,  in  the  case  of  the 
trained  group,  the  USAICS  course. 


Objectives  of  the  Present  Research 

The  present  experiment  sought  (1)  to  establish  the  underlying 
conceptual  structure  which  determines  the  way  analysts  evaluate  the 
quality  of  intelligence  data,  (2)  to  examine  the  relationship  of  current 
Reliability /Accuracy  ratings  to  the  dimensions  of  the  underlying 
conceptual  structure,  and  (3)  to  examine  the  utility  of  the  conceptual 
structure  used  to  organize  intelligence  data.  The  question  of  the 
amount  and  type  of  information  needed  to  make  an  accurate  assessment  of 
the  quality  of  intelligence  data  was  not  addressed. 


METHOD 

The  approach  is  based  in  techniques  and  procedures  of  numerical 
taxonomy  (Sneath  and  Sokal,  1973)*,  which  have  been  used  widely  in 
biological  analysis  to  identify  natural  grouping  of  species  empirically. 
Here,  these  techniques  identify  natural  groupings  or  families  of  messages 
and  determine  the  characteristics  considered  Important  by  observers  in 
evaluating  the  messages. 


*Sneath,  P.  H. , and  Sokal,  R.  R.  Numerical  Taxonomy.  San  Francisco; 
W.  H.  Freeman  & Co.,  1973. 
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The  intelligence  analyst's  task  was  subdivided  into  basic  judgments 
about  the: 

(1)  reliability  of  the  data  source  (or  agency) ; 

(2)  accuracy  of  the  information; 

(3)  situation  (its  stability,  and  the  magnitude  of  the  action  or 
size  of  forces  involved); 

(4)  action  reported  (feasibility  of  the  action,  and  risks  for  the 
rater's  unit  inherent  in  such  action); 

(5)  inferences  to  be  drawn  ultimately  from  the  information  (the 
value  or  utility  of  the  information,  and  the  nature  of  the  Inferences 
which  can  be  attributed  to  it);  and 

(6)  characteristics  of  the  message  itself  (clarity,  timeliness, 
comprehensiveness) . 

Taken  together,  the  aspects  of  incoming  information  define  an  a priori 
conceptual  framework  that  describes  the  data  supplied  to  analysts. 

These  aspects  also  suggested  specific  bipolar  measurement  scales  or 
features,  shown  in  Table  1. 

The  discriminatory  power  of  the  taxonomic  characteristics,  their 
mutual  independence,  and  their  adequacy  in  representing  the  judgmental 
process  is,  of  course,  subject  to  empirical  test.  These  scale 
dimensions  do  not  exhaust  the  ways  in  which  intelligence  information 
might  be  evaluated. 

However,  the  aspects  of  intelligence  evaluation  chosen  for  measure- 
ment were  judged  to  be  sufficiently  extensive  to  enable  participants 
to  make  a comprehensive  evaluation  of  a selected  set  of  tactical 
messages  while  keeping  the  rating  task  within  reasonable  and  practicable 
bounds . 

The  46  characteristics  or  qualities  shown  in  Table  1 represent 
judgments  that  depend  on  an  interpretation  of  the  content  of  the 
messages.  As  such,  they  cannot  be  considered  strictly  as  characteristics 
of  the  stimulus.  Instead,  they  represent  in  abstract  terms  the  manner 
in  which  the  analysts  make  their  judgments  of  message  content. 

Thus,  the  rating  scales  serve  as  the  operational  definition  of  the 
conceptual  framework  of  the  structure  of  intelligence  messages;  the 
empirical  analyses  serve  to  test  this  framework  and  thereby  provide  a 
more  precise  understanding  of  an  analyst's  message  ratings. 
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Table  1.  Judgmental  Aspects  of  Intelligence  Analysis 


Basis  for  Judgment 

Elements  of  Judgment 

Qualities  Rated  by 

Bipolar  Scales 

Source/Agency 

Reliability 

Reliable-Variable 

Dependab le -Und ependab le 
Truthful-Deceptive 
Acceptable-Unacceptable 
Faultless-Faulty 

Information 

Accuracy 

Accurate-Erroneous 

True-False 

Confirmed-Unconfirmed 

Consistent-Inconsistent 

Subs tan tiated-Contradicted 

Situation 

Stability 

Stable -Unstable 

Constant-Changing 

Precarious-Imprecarious 

Inert-Volatile 

Active-Inactive 

Magnitude 

Large  Scale-Small  Scale 

Widespread-Local 

Massive-Insignificant 

Many- Few 

Heavy-Light 

Action 

Feasibility 

Feasible-Infeasible 
Possible- Impossible 

Probab le- Improbable 

Likely-Unlikely 

Expected-Unexpected 

Risk 

Routine-Risky 

Ordinary-Extraordinary 

Safe-Dangerous 

Secure- Insecure 
Unhazardous-Hazardous 

Inference 

Utility 

Useful-Useless 
Relevant-Irrelevant 
Pertinent-Extraneous 
Analyzable-Unanalyzable 
Appropria  te - Inappropriate 

Attribution 

Observed-Inferred 
Unimplied-Implied 
Uninterpreted- Interpreted 
Factual- Theoretical 
Direct-Indirect 

Report 

Comprehension 

Specific -Vague 

Clear-Garbled 

Intelligible-Unintelligible 

Understandable-Confusing 

Timeliness 

Timely-Untimely 

Comprehensiveness 

Fragmented- Complete 

— J 
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Model  for  Data  Collection 

Ordinarily,  in  biology  or  linguistic  semantics,  the  presence  or 
absence  of  a particular  trait  or  characteristic  is  tabulated  in  a matrix 

which  takes  the  form  of  Figure  1 where  the  represents  groupings  of 

characteristics,  f ^ , and  each  datum  is  described  by  a pattern  of  fj 

occurrences,  +,  or  absences, 


DATA  ELEMENTS 

12  3 4 5 


Figure  1.  Two-dimensional  Taxonomic  Data  Matrix 


In  order  to  provide  for  greater  refinement  of  measurement,  the  bipolar 
qualities  have  been  scaled  in  seven  steps  indicating  degrees  of  intensity. 
Thus,  the  feature  matrix  becomes  three-dimensional  (Figure  2). 


Figure  2.  Three-dimensional  Taxonomic  Data  Matrix 


If  we  now  add  multiple  observers  as  judges  of  each  feature  of  each 
datum,  the  matrix  becomes  four- dimens iona 1 . Collapsing  the  structure 
over  degrees  of  intensity  (di)  of  quality  scales,  for  the  purpose  of 
illustration,  produces  the  taxonomic  data  matrix  shown  in  Figure  3 where 
1 < d^  < 7 represents  the  scalar  degree  of  each  set  of  bipolar  qualities 
for  each  datum  (message). 


Figure  3.  Four-dimensional  Taxonomic  Data  Matrix 
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The  vise  of  multiple  observers  in  this  model  represents  the  repeti- 
tion of  the  judgmental  process  to  enhance  the  reliability  of  each 
judgment.  This  procedure  allows  for  averaging  of  observation  sampling 
errors  and  permits  the  use  of  parametric  indices  of  message  similarity. 

It  is  possible  to  construct  separate  observer  sampling  divisions 
within  the  total  set  of  observers  so  as  to  assess  the  commonality  of  the 
judgments  across  separate  groups.  Such  sampling  divisions  were  employed 
in  the  present  project. 


Data  Elements 

The  basic  taxonomic  units  or  data  elements  to  be  evaluated  by  the 
analysts  were  33  naturally  occurring  intelligence  messages  drawn  from 
reports  of  enemy  activity  during  the  period  10  December  through  15 
December  1944  along  the  Ardennes  line  in  Europe.  To  these  33  messages, 
seven  fictitious  and  misleading  reports  were  added  to  determine  whether 
analysts  would  detect  them.  Thus,  each  subject  would  evaluate  40 
messages . 

These  messages  (Appendix  A)  depicted  the  buildup  of  the  defending 
German  Army  which  eventually  culminated  in  an  aggressive,  large  scale 
counteroffensive  and  breakthrough  of  the  Allied  lines  on  16  December, 
known  familiarly  as  the  Battle  of  the  Bulge.  After  the  fact,  it  is 
possible  to  deduce  the  significance  of  the  intelligence  messages. 

At  the  time,  however,  the  strength  of  the  attack  by  a presumably 
defeated,  demoralized  and  hard-pressed  enemy  caught  the  Allies  by 
surprise.  The  messages  vary  from  trivial  sightings  of  horse-drawn 
carts  to  reports  of  massive  troop  movements,  POW  interrogations,  and 
G2  summaries.  They  vary  in  length  from  short  to  long,  differ  in  degree 
of  completeness,  and  originated  from  several  agencies  and  sources. 

Although  the  situation  is  dated  and  involves  a style  of  warfare 
which  differs  sharply  from  the  counterinsurgencies  of  more  recent  times, 
the  messages,  unlike  fictional  scenarios,  are  real,  have  a documented 
outcome,  and  are  naturallstically  valid. 
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Taxomonic  Analysis 

A clear  definition  of  the  data  elements  or  "operational  taxonomic 
units"  of  interest  and  a conceptual  framework  specifying  which  charac- 
teristics of  the  data  elements  should  be  measured  provide  the  basic 
elements  for  a taxonomic  analysis.  The  present  analyses  focused  on  the 
relationships  among  the  characteristics  of  the  messages. 

Any  consistent  patterns  in  the  use  of  the  rating  scales  across 
observers  and  across  messages  gave  clues  about  the  nature  of  the  under- 
lying judgment  space.  Thus,  if  observers  consistently  rated  as 
"probable"  and  "likely"  messages  which  they  had  also  said  were  "possible," 
then  one  dimension  of  their  perceptual  framework  was  some  concept  of  the 
likelihood  of  the  event  reported. 

In  a similar  fashion,  similarities  among  messages  were  examined. 

Some  messages  were  consistently  rated  as  "likely,"  others  as  "unlikely," 
some  as  "routine,"  others  as  "risky."  By  grouping  together  messages 
which  received  similar  ratings,  natural  families  of  messages  were 
identified. 
iJ 

The  same  data  were  used  to  examine  both  the  judgment  space  (the 
grouping  of  characteristics)  and  the  report  space  (the  grouping  of 
messages).  If  our  original  conceptual  framework  was  valid,  then  we  would 
expect  that  the  grouping  of  messages  would  be  maaningful  to  intelligence 
analysts . 

However,  complete  validation  of  the  framework  would  require  a revision 
of  the  rating  scales  followed  by  data  collection  and  examination  of 
the  message  groupings  using  the  revised  scales. 

Rating  Scales 

The  subjects  rated  each  of  the  40  messages  on  the  46  quality  scales 
chosen  to  represent  the  elements  of  judgment  identified  in  the  concept 
of  the  rater's  task,  as  shown  in  Table  1.  ’ In  addition,  the  traditional 
Reliability  and  Accuracy  rating  scales  and  a Global  or  overall  validity 
scale  were  included  in  the  rating  task. 

This  latter  scale,  suggested  by  Samet  (1975),  was  used  twice,  at  the 
beginning  and  end  of  the  rating  task  for  each  message.  Figure  4 shows 
the  scales  as  presented  on  the  subjects'  response  form;  the  second 
Global  rating  was  on  the  reverse  side  of  the  form. 


e 4.  Rating  Scales 
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Subjects 

The  sample  employed  in  this  project  consisted  of  a school-trained 
group,  hereafter  referred  to  as  "trained,"  and  one  designated  as 
"untrained."  Subjects  were  59  enlisted  personnel  assigned  to  the  course 
for  intelligence  analysts  (96B)  at  the  U.S.  Army  Intelligence  Center  and 
School  (USAICS) , Fort  Huachuca,  Arizona. 

The  untrained  group  had  just  enrolled  in  the  school  and  the  school- 
trained  group  had  just  completed  the  training  course.  Of  the  59  subjects 
tested,  the  data  from  four  were  discarded  on  grounds  of  incompleteness  or 
obvious  pattern  checking  behavior  (e.g.,  rating  all  messages  haphazardly). 
The  final  complement  of  55  subjects  consisted  of  34  untrained  and  21 
trained  analysts  (Table  2).  Both  groups  had  only  minimal  Army  experience. 

Test  Procedure 

Subjects  were  provided  a 1:100,000  map  of  the  area  in  which  the 
action  took  place,  with  troop  deployments  as  of  10  December  1944  over- 
printed on  the  map.*  An  acetate  overlay  and  grease  pencil  were  provided 
each  subject  for  making  notes  and  changing  deployments  as  each  intelli- 
gence report  was  examined. 

In  addition,  the  subjects  were  given  a book  of  instructions  on  the 
task,  a book  of  answer  forms,  and  the  set  of  sequentially  ordered  intelli- 
gence reports  covering  the  period  from  10  December  through  15  December. 

Each  group  was  tested  in  a session  of  approx  .aately  three  and  one- 
half  hours.  Two  monitors  were  present  during  the  sessions  to  give 
instructions,  answer  questions,  and  check  on  progress  throughout  the 
data  collection. 


ANALYSIS 

Data  Reduction 

The  data  comprised  110,000  observations  collected  from  the  55 
subjects  who  rated  each  of  the  40  reports  on  the  50  scales.  The 
resulting  raw  data  matrix  was  examined  from  a number  of  viewpoints 
(Figure  5).  All  data  were  transferred  to  punch  cards,  and  the  analyses 
proceeded  according  to  the  five  steps  indicated  in  the  figure. 


*A  USAICS  Practical  Exercise  for  nonresident  instruction  is  based  on  the 
same  area.  Prepared  maps  were  thus  available  and  provided  by  the 

Intelligence  School. 
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Table  2.  Characteristics  of  the  Sample 


Characteristics 

Trained 

Untrained 

Total 

21 

34 

Men 

16 

27 

Women 

5 

7 

Grade: 

E1-E5 

17 

33 

E6-E9 

4 

1 

Length  of  Service: 

1-6  months 

13 

28 

7-12  months 

2 

3 

13-18  months 

1 

0 

19-24  months 

0 

0 

25  + months 

5 

3 

Previous  Intelligence  training: 

School 

0 

0 

On-the-Job  training 


0 
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The  analyses  compared  the  trained  and  untrained  subjects  using  data 
averaged  over  the  separate  reports  to  obtain  mean  scale  ratings  for  each 
subject,  or  averaged  over  scales  to  obtain  mean  report  ratings  for  each 
subject.  Each  type  of  analysis  is  interpreted  in  the  following  sections. 
Only  summary  tables  of  these  analyses  are  presented.* 

These  analyses  were  designed  to  answer  the  following  questions,  which 
are  keyed  to  the  analysis  steps  of  Figure  5: 

(I)  Is  the  pattern  of  quality  ratings  employed  by  the  trained  and 
untrained  analysts  discriminably  different?  In  what  ways,  if  any,  does 
the  school  training  make  a difference  in  the  conceptual  framework  used? 

(II)  What  is  the  nature  of  the  conceptual  framework  underlying  the 
evaluation  of  intelligence  reports?  Is  there  an  aggregate  structure 
common  to  trained  and  untrained  subjects? 

(Ill)  What  is  the  nature  of  the  resulting  report  classification? 

How  were  reports  grouped  by  the  subjects  when  the  scale  qualities  are 
analyzed?  Is  the  report  structure  meaningful?  Does  it  imply  measure- 
ment attributes  for  which  there  is  no  current  measurement  or  training? 

(IV)  What  are  the  determinants  of  the  currently  employed  Reliability 
and  Accuracy  judgments?  Are  these  judgments  independent  across  differing 
reports  and  subject  groups?  Can  they  be  assessed  by  different  and 
perhaps  purer  methods? 

(V)  What  is  the  predictive  validity  of  the  Accuracy  and  Reliability 
ratings  with  respect  to  individual  subjects?  Is  the  validitv  of  these 
scale  ratings  influenced  by  subject  training?  Does  the  predictability 
of  the  ratings  differ  with  respect  to  subject  or  report  variation. 


Discriminant  Function  Analysis-  Subject  Groups 


The  untrained  group  of  students  were  included  in  the  experiment  to 
provide  data  which  would  facilitate  by  contrast  the  interpretation  of 
results  from  the  trained  group.  Given  the  two  groups,  their  use  of  the 
rating  scales  could  be  directly  compared. 

For  this  analysis,  averages  of  each  rating  by  all  subjects  were  taken 
over  the  40  intelligence  reports  and  examined  to  determine  whether  the 
trained  and  untrained  subjects  could  be  discriminated  by  a linear 
discriminant  function  based  on  the  patterns  of  their  respective  uses  of 
the  individual  rating  scales. 


*Full  data  printouts  of  each  analysis  may  be  examined  at  ARI. 
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Figure  5.  Data  Analysis  Procedures 
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On  the  average,  across  all  reports.  If  any  given  individual  tended 
to  employ  rating  scales  so  as  to  produce  a pattern  of  ratings  like  that 
of  other  subjects  in  his  designated  group  and  different  from  that  of 
subjects  in  the  contrasting  group,  discrimination  would  be  achieved. 

The  test  of  discrimination  is  made  on  the  basis  of  the  a posteriori 
assignment  of  the  subject  to  one  or  the  other  group  on  the  basis  of  the 
calculated  regressions.  The  two  groups  discriminant  case  is  equivalent 
in  all  regards  to  a multiple  regression  analysis  using  a dummy 
dichotomous  variable  as  the  criterion. 

Table  3 details  the  15  steps  required  to  achieve  100%  discrimination 
between  the  two  groups  of  subjects.  Each  step  of  the  discriminant 
analysis  represents  the  addition  of  a new  rating-scale  variable  to  the 
linear  discriminant  or  regression  equation. 

The  choice  of  scale  variable  at  each  successive  step  is  made  on  the 
basis  of  the  maximum  amount  of  discrimination  which  can  be  achieved 
from  the  data  supplied.  Thus,  at  the  first  step,  the  average  rating-scale 
judgments  of  the  two  groups  for  the  message  quality  PRECARIOUS-IMPRECARIOUS 
provided  the  greatest  amount  of  discrimination  between  the  two  groups. 

At  this  step,  21  of  the  34  untrained  subjects  were  correctly  classi- 
fied as  members  of  that  group  and  18  of  the  21  trained  subjects  were 
correctly  classified  as  members  of  the  trained  group.  At  step  2,  the 
next  scale  quality,  ANALYZABLE-UNANALYZABLE , discr iminatively  moved 
three  additional  subjects  from  misclassif ication  as  trained  subjects  to 
the  correct  classification  of  untrained.  However,  five  of  the  previ- 
ously correctly  classified  trained  subjects  were  lost  to  misclassif ication 
as  untrained.  However,  by  the  time  the  regression  equation  contained 
the  15  rating  scales  shown  in  Table  3,  all  subjects  had  been  correctly 
classified . 

On  the  basis  of  these  variables,  a consistent  picture  emerged  of  the 
difference  between  these  groups.  The  trained  analysts  judged  the 
reports  with  much  less  alarm  and  as  more  routine  than  did  the  untrained 
analysts,  at  the  same  time  making  finer  judgments  of  the  form  of  the 
report.  The  trained  analyst  was  better  able  to  analyze  the  content  and 
import  of  the  reports  and  make  better  judgments  of  threat  potential  and 
accuracy.  Additionally,  there  was  much  wider  variation  in  the  average 
judgments  of  the  accuracy  and  truthfulness  of  these  reports  among  the 
trained  subjects,  indicating  a greater  degree  of  discrimination. 
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Table  3*  Discriminants  of  Quality  Judgments  between  Trained  and  Untrained  Analysts 
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There  are  implications  for  training  in  the  differences  between 
groups.  The  trained  and  untrained  groups  were  discriminably  different 
in  their  use  of  the  rating  scales,  and  the  pattern  of  these  differences 
is  consistent  with  that  which  should  result  from  an  effective  training 
course  in  intelligence  analysis.  However,  as  will  be  seen  below,  the 
rating  scales  that  were  prominent  in  the  discriminant  function  were 
generally  not  those  which  are  an  important  part  of  the  subjects'  judgment 
space.  These  results  alone  do  not  provide  a sufficient  basis  either 
for  the  evaluation  of  the  96B  training  course  or  for  changes  in 
instruction  relative  to  the  evaluation  of  intelligence  data.  However, 
the  differences  noted  suggest  areas  where  current  training  does  have  an 
impact.  Thus,  the  trained  subjects  were  much  more  aware  of  the  signifi- 
cance of  messages  (or  the  lack  of  significance) , presumably  through 
their  exposure  to  a number  of  practical  exercises  where  they  had  to  deal 
with  such  messages  in  the  context  of  tactical  situations.  However,  this 
awareness  may  reflect  amount  of  experience  more  than  amount  of  intelli- 
gence training  per  se.  The  trained  subjects  also  were  different 
(although  not  necessarily  better)  in  their  use  of  the  standard  Accuracy 
rating. 


Factor  Structure  of  the  50  Basic  Scales 

Average  values  for  ratings  by  the  trained,  untrained,  and  combined 
groups  on  each  quality  scale  were  obtained  for  each  report.  These  means 
factor,  on  the  average,  cannot  be  used  to  predict  the  size  of  its 
ratings  with  respect  to  other  factors.  These  means  can  be  considered  to 
be  the  best  overall  assessment  of  each  of  the  reports  with  respect  to 
each  characteristic.  The  structure  of  the  patterns  of  similar  scale 
usage  derived  from  these  averages  provided  an  assessment  of  the  implicit 
conceptual  space  within  which  each  of  the  subject  groups  evaluated  the 
reports,  reflecting  the  resolving  power  of  the  analysts  and  the  qualities 
to  which  they  are  responsive.  The  conceptual  structure  can  be  compared 
to  the  discrimination  and  bandwidth  capabilities  of  a special  purpose 
radio  receiver.  Since  the  qualities  were  chosen  to  span  the  range  of  all 
abstract  potential  qualities  of  intelligence  reports,  the  resultant 
structure  can  be  said  to  indicate  the  selective  tuning  which  character- 
izes the  analyst  receiver. 

Principal  components  factor  analysis  was  the  specific  technique 
used  to  describe  the  underlying  dimensions  in  the  judgments.  Subsequent 
varimax  rotation  of  the  structure  used  in  these  analyses  had  the  effect 
of  relocating  the  factor  dimensions  in  the  judgmental  space  so  as  to 
spread  the  factor  variance  more  evenly  across  the  separate  factors.  This 
technique  also  maximizes  the  within-factor  variance  and  discards  small, 
specific,  or  error  factors.  Each  separate  dimension  must  conform  to  the 
constraint,,  after  rotation,  that  it  be  perpendicular  to  every  other 
dimension.  That  is,  the  resultant  factor  space  represents  a cartesian 
space  of  independent  dimensions.  Variation  within  one  factor  is  inde- 
pendent of  variation  in  every  other  factor;  the  judgmental  ratings  of  a 
variable  with  respect  to  one  factor,  on  the  average,  cannot  be  used  to 
predict  the  size  of  its  ratings  with  respect  to  other  factors. 
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Table  4 displays  Che  mean  values,  for  Che  Chree  subjecc  groups,  of 
Che  50  quallcles  used  as  Che  variables  in  Chese  analyses.  Scales  1 
Chrough  46  vary  from  1 chrough  7,  Che  ReliabiliCy  and  Accuracy  racings 
(scales  47  and  48)  vary  becveen  1 and  6,  and  Che  cwo  Global  racings 
(scales  49  and  50)  vary  from  0 chrough  1.00 

UnCrained  Subjects.  Table  5 displays  Che  scale  facCor  structure  for 
uncrained  subjecCs.  The  "facCor  coef ficienCs"  lisCed  are  che  "loadings" 
of  che  scales  on  cheir  respecCive  facCors.  The  absoluCe  values  of  Che 
loadings  reflecC  che  reladve  importance  of  che  scales  as  represenCacives 
of  che  underlying  judgmenC  dimension  being  used  by  Che  subjecCs. 

All  scales  have  some  loading  on  all  facCors;  lisdng  of  scales  by 
facCors  as  in  Table  5 involves  a somewhaC  arbicrary  cuCoff  ac  che  poinC 
where  ic  appears  chaC  any  addiclonal  scales,  because  of  Cheir 
relacively  low  loadings  on  Che  facCor,  would  conCribuce  liccle  Co  an 
undersCanding  of  Che  underlying  judgmenC  dimension  represenced  by  Che 
facCor. 

Some  scales  are  "reflecCed"  Co  be  parallel  wich  ocher  scales  on  a 
facCor;  chus  a score  of  1 on  Che  False-True  scale  may  be  represenced  as 
a 7 on  che  True- False  scale,  a score  of  2 as  a 5 on  Che  reflecced  scale, 
and  so  on.  This  procedure  minimizes  Che  number  of  minus  signs, 
simplifies  incerprecacion,  and  in  no  way  affecCs  che  meaning  of  Che 
resulcs. 

Noce  ChaC  Che  Global  Validicy  scales  have  noC  been  reflecced;  a high 
number  on  Chese  scales  represenCs  a racing  of  high  Global  Validicy, 
while  a low  number  on  che  cradicional  Accuracy  and  ReliabiliCy  scales 
represenCs  high  Accuracy  or  ReliabiliCy. 

The  four  roCaCed  facCors  of  Che  sCrucCure  in  Table  5 accounC  for 
85%  of  che  CoCal  report  variance  in  racings.  FacCor  I accounCs  for 
54%  of  Che  variance,  consisCs  of  scales  represenCing  judgmencs  of 
accuracy,  reliabilicy,  and  CruCh,  and  is  labeled  ACCURACY. 

FacCor  II  accounCs  for  21%  of  Che  variance  and  involves  Che  ChreaC 
potenClal  and  CacCical  relevance  of  reports.  FacCor  II  Includes 
aspecCs  of  Che  subjecCs'  evaluaCions  of  Che  reporCs  dealing  wich  Che 
magnlCude  and  import  of  che  developing  CacCical  sicuacion  and  is 
labeled  RELEVANCE. 

FacCor  II  is  very  similar  Co  facCors  chac  have  emerged  in  ocher 
research  involving  qualify  Judgments  of  action  stimuli.  Therein,  this 
factor  represenced  a combination  of  the  activity  and  potency  of  the 
stimuli  and  was  labeled  Dynamism. 
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Table  4.  Group  Means  and  Standard  Deviations  of 
Combined  Reports,  by  Rating  Scale 
(Page  1 of  2) 


Untrained  Trained  Combined 


Scale 

Mean 

SD 

Mean 

SD 

Mean 

SD 

1 

(Active) 

2.84 

0.68 

2.58 

O.67 

2.74 

O.63 

2 

( False) 

5-45 

0.57 

5-71 

0.83 

5-55 

O.63 

3 

(Untime ly) 

5-09 

0.56 

5-33 

0.50 

5.18 

0.50 

4 

(Specific) 

3.00 

O.55 

3.06 

O.67 

3.02 

0.52 

5 

( Precarious) 

3-51 

0.37 

4 .05 

0.29 

3.72 

0.29 

6 

( Extraordinary) 

3-79 

0.54 

4.35 

0.60 

4 .01 

0.52 

7 

( Intelligible) 

2-77 

0.39 

2.41 

0.40 

2.63 

0.34 

8 

(infeasible) 

5-21 

0.43 

5.61 

0.6l 

5.36 

0.44 

9 

( Factual) 

2.98 

0.46 

2.67 

0.64 

2.86 

0.48 

10 

(Unstable) 

4.72 

0.50 

4 .72 

0.59 

4.72 

0.48 

11 

(Direct) 

2.95 

0.48 

2.93 

0.46 

2.94 

0.43 

12 

(Variable) 

4.67 

O.54 

5.12 

0.88 

4.84 

0.62 

13 

( Under s tandab le ) 

2-77 

0.44 

2.64 

O.53 

2.72 

0.41 

14 

(Like ly) 

2.67 

0.48 

2.43 

0.66 

2.58 

0.50 

15 

(Massive) 

3-28 

0.70 

3.23 

0.58 

3.26 

0.62 

l6 

(Contradicted) 

4 .73 

0.44 

4 .65 

0.48 

4.70 

0.41 

17 

( Dependable) 

2.83 

0.65 

2.64 

0.96 

2.76 

0.73 

18 

(Useless) 

5-17 

0.57 

5-53 

0.46 

5-31 

0.48 

19 

(inert) 

4.08 

0.36 

4.02 

0.33 

4.05 

0.28 

20 

( Inappropriate) 

5.00 

0.44 

5.23 

0.43 

5-09 

0.38 

21 

(Truthful) 

3.79 

0.6l 

2.65 

O.83 

2.74 

0.66 

22 

(Unhazardous) 

4 .74 

0.47 

4.48 

0.38 

4.64 

0.39 

23 

(Expected) 

3.91 

0.37 

3.52 

0.46 

3.76 

0.37 

24 

(Unimplied) 

4.48 

0.29 

4.37 

0.34 

4.44 

O.23 

25 

(Heavy) 

3.58 

0.76 

3.85 

0.47 

3.68 

0.6l 

26 

( Possible) 

2.42 

0.42 

2.10 

0.48 

2.30 

0.40 

27 

(Observed) 

2.96 

O.87 

2.93 

0.99 

2.95 

0.87 

28 

(Garbled) 

5.30 

0.45 

5-24 

0.49 

5.28 

0.43 
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Table  4 
(Page  2 of  2) 


Scale 

Untrained 
Mean  SD 

Trained 
Mean  SD 

Combined 
Mean  SD 

29 

( Faultless) 

3.39 

0.51 

3-42 

0.63 

3-40 

0.51 

30 

(Widespread) 

3.87 

0.69 

4.15 

0.57 

3.98 

0.62 

31 

(Accurate) 

2.86 

0.57 

2.69 

0.74 

2.80 

0.60 

32 

(Uninterrupted) 

4.37 

0.37 

4.31 

0.44 

4.35 

0.28 

33 

(Relevant) 

2.87 

0.49 

2.44 

0.51 

2.71 

0 .46 

34 

(insecure) 

4.34 

0.43 

4.63 

0.50 

4 .45 

0.41 

35 

(Analyzable) 

2.75 

0.49 

2.26 

0.42 

2.56 

0.42 

36 

(Risky) 

3.33 

O.56 

3.86 

0.46 

3-53 

0.48 

37 

(Fragmented) 

4.42 

0.44 

4.55 

0.48 

4 .47 

0.40 

38 

(Acceptable) 

2.78 

O.52 

2.64 

0.66 

2-73 

0.53 

39 

(Many) 

3.58 

0.73 

3.83 

0.72 

3.68 

O.^O 

40 

(Confirmed) 

3.37 

0.66 

3 .68 

0.93 

3-49 

0.73 

4l 

(Changing) 

3-89 

0.31 

3.76 

0.36 

3.84 

0.27 

42 

(Large  Scale) 

5.57 

0.87 

3.87 

0.81 

3.68 

0.82 

43 

(improbable) 

5.13 

0.47 

5-43 

0.66 

5.25 

0.49 

44 

(Extraneous) 

4.58 

0.33 

5-34 

0.46 

4.87 

0.34 

45 

(Consistent) 

3.24 

0 .46 

3.15 

0.49 

3-12 

0.4l 

46 

(Dangerous) 

3.28 

0.50 

3.54 

0.39 

3.38 

0.42 

47 

( Reliability) 

2.53 

0.57 

2.48 

1.52 

2.51 

O.89 

48 

(Accuracy) 

2.47 

0.50 

2.70 

1.09 

2.56 

O.69 

49 

(Global-1) 

O.74 

0.08 

0.80 

0.14 

0.77 

0.10 

50 

(Global-2) 

0.75 

0.11 

0.8l 

0.14 

0-77 

0.11 

Note.  The  label  for  the  bipolar  scales  indicates  the  low  end  of  the  1-7  scale. 
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Factor  III  accounts  for  7%  of  the  variance  and  seems  to  reflect  a 
Judgment  about  applicability  of  message  content,  a determination  whether 
the  content  could  be  used  directly  or  required  drawing  some  inference. 
This  factor  was  tentatively  labeled  DIRECTNESS.  Factor  IV  is  a relatively 
unimportant  specific  factor,  accounting  for  only  3%  of  the  variance,  and 
will  be  omitted  in  subsequent  discussion. 

Trained  Subjects.  Table  6 presents  the  rotated  structure  for  the 
trained  subjects.  The  structure  accounts  for  73%  of  the  total  report 
rating  variance.  The  distribution  of  the  variance  across  the  four 
factors  is  more  uniform  than  that  found  for  untrained  subjects,  implying 
that  the  trained  subjects  were  using  finer  discriminations  in  their 
Judgments  of  the  attributes  of  the  AO  reports. 

Whereas  the  untrained  subjects  showed  a strong  tendency  to  confine 
their  evaluations  to  Judgments  of  the  accuracy  of  the  reports,  the 
trained  subjects  found  other  discriminable  qualities  on  which  to  base 
their  Judgments.  As  before,  the  most  crucial  Judgmental  dimension  was 
ACCURACY;  RELEVANCE  slipped  down  in  importance  from  second  to  a close 
third. 

The  third  factor  extracted  is  best  labeled  DIRECTNESS  because  of  the 
scales  contributing  to  it  (Expected,  Ordinary,  Feasible),  although  for 
trained  subjects  it  includes  INTELLIGIBILITY  AND  USEFULNESS  as  corre- 
lated attributes.  Nonetheless,  the  factor  is  essentially  similar  to 
that  found  for  the  untrained  subjects. 

Factor  IV  does  not  provide  the  basis  for  straightforward  interpre- 
tation. Although  the  inference-contributing  determinant  of  IMPLIED  has 
the  greatest  influence  on  this  factor,  the  miscellaneous  nature  of  other 
qualities  such  as  specificity  and  fragmentation  makes  it  difficult  to 
assign  an  appropriate  label.  Since  it  contributes  only  6%  to  the  total 
structure  and  is  heterogeneous  in  nature,  it  was  dropped  from 
consideration. 

Combined  Subject  Groups.  Table  7 presents  the  composite  structure 
for  the  total  subject  sample.  Since  the  separate  group  structures  are 
highly  similar  in  factor  content  and  prominence,  this  structure  reflects 
the  common  perceptual  space  within  which  analysts  in  general  can  be 
expected  to  perform. 

Synthesis  of  scales  contributing  to  the  factors  confirmed  ACCURACY 
as  an  appropriate  label  for  Factor  I,  RELEVANCE  for  Factor  II,  and 
DIRECTNESS  for  Factor  III.  A fourth  factor  contributing  only  3%  to  the 
variance  was  not  considered  further. 
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The  dimensions  of  the  factor  structure  represent  independent  aspects 
of  the  judgmental  process  involved  in  intelligence  analysis.  They  are 
the  critical  underlying  determinants  of  the  judgment  space  in  which  the 
analyst  makes  his  discriminations  among  the  message  he  is  asked  to 
evaluate . 

Neither  the  Reliability  nor  the  Accuracy  rating  scales  as  currently 
employed  adequately  captures  the  dimensionality  of  this  space.  Both 
these  rating  procedures  assess  only  the  first  axis  of  that  space — 
ACCURACY;  ether  scales  are  more  representative  of  the  underlying  ACCURACY 
dimension.  Considerably  greater  discriminating  variability  in  the 
reports  is  left  entirely  unassessed  by  current  measures. 

Although  the  judgment  space  for  the  two  groups  of  subjects  is  nearly 
identical  in  composition,  this  similarity  does  not  imply  that  the  two 
groups  evaluated  the  reports  identically.  It  is  the  structural  space 
within  which  those  evaluations  were  made  that  is  the  same. 

Thus,  both  groups  considered  important  the  information  qualities  of 
accuracy,  relevance,  and  directness.  Specific  evaluations  of  the 
quality  of  a given  message  by  the  two  groups  are,  however,  discriminably 
different,  as  has  been  established. 

The  situation  is  analogous  to  that  of  two  observers  who  can  see  the 
same  visual  stimulus  but  who  differ  as  to  the  significance  or  interpre- 
tation of  that  stimulus.  No  matter  how  much  training  an  observer  is 
given,  he  cannot  make  accurate  discriminations  among  stimulus  objects 
to  which  his  perceptual  makeup  is  insensitive. 

The  significance  of  these  findings  concerning  the  scale  factor 
structure  is  that  the  analyst  comes  to  his  task  with  an  implicit 
perceptual  structure.  To  be  maximally  effective,  training  instructors 
must  understand  and  capitalize  upon  that  implicit  structure.  Findings 
of  the  present  analysis  strongly  suggest  that  this  structure  is  not 
modified  by  the  Army  Intelligence  School  training  procedures. 

In  fact,  what  is  known  of  such  implicit  perceptual  structures 
indicates  that  it  would  be  extremely  difficult  to  modify  that  structure. 
What  is  required  is  that  the  training  use  the  underlying  conceptual 
structure  to  best  advantage. 


Factor  Structure  of  Intelligence  Reports 

For  the  analyses  of  the  structure  of  the  report  space,  the  mean 
subject  data  matrices  of  the  preceding  analyses  were  transposed  so  that 
the  pattern  of  intercorrelations  among  reports  could  be  examined  with 
respect  to  scale  means.  Analyses  of  the  report  groupings,  as 
determined  by  the  analysts'  judgments,  reflect  the  degree  to  which  the 
subjects'  judgment  space  served  to  differentiate  the  intelligence 
reports . 
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The  analyses  represent  a form  of  internal  validity  test  of  the  scale 
structure  obtained.  If  this  grouping  of  reports,  this  taxonomv,  is 
Judged  meaningful,  significance  of  the  scale  structure  from  which  derived 
would  be  confirmed.  The  report  structure  was  included  for  this  purpose. 

Table  8 shows  the  report  means  for  the  three  subject  groups  across 
all  scales.  All  the  means  are  substantially  equivalent,  and  the  variance 
about  those  means  is  substantially  higher  than  that  found  for  the  scale 
means  of  Table  4.  The  explanation  is  that  these  analyses  have  taken 
averages  over  what  is  already  known  to  be  a factorially  diverse  and 
independent  set  of  scales.  The  averages  displayed  reflect  only  the 
average  positional  biases. 

Lack  of  discriminability  in  the  report  averages,  however,  does  not 
indicate  sameness  among  the  reports  with  respect  to  individual  scale 
ratings  to  be  factored.  On  the  contrary,  the  high  variances  about  these 
means  indicate  considerable  differences  in  scale  ratings  from  report  to 
report . 

Analyses  have  a distinct  mathematical  dependency  on  those  previouslv 
done.  The  report  structures  necessarily  cannot  contain  greater 
discriminability  than  emerges  from  analyses  of  the  scale  structure. 
Technically,  that  is,  the  rank  of  the  report  structure  matrix  is 
determined  by  the  rank  of  the  judgment  space,  which  has  been  found  not 
to  exceed  3.  The  reports  grouped  together  on  the  factors  described 
below  are  combined  in  terms  of  a common  variance  on  ACCURACY,  RELEVANCE 
and  DIRECTNESS . Later  displays  make  this  picture  clearer  by  modeling 
the  report  distribution  within  the  perceptual  space. 

Tables  9 through  11  detail  the  report  groupings  by  factors  that 
emerge  from  these  analyses.  Structures  for  the  three  groups  of  untrained, 
trained,  and  combined  subjects  are  quite  similar,  and  discussion  is 
focused  on  the  combined  structures  shown  in  Table  11. 

However,  some  differences  between  the  groups  were  noted:  The 
trained  subjects  used  only  a two-dimensional  structure  to  group  the 
messages.  The  report  structure  for  the  trained  subjects  in  Table  10 
shows  that  all  messages  deriving  from  friendly  and  reliable  sources  have 
high  loadings  on  Factor  I,  and  all  messages  based  on  POW,  civilian,  and 
deserter  sources  have  high  loadings  on  Factor  II. 

Considering  the  fact  that  ACCURACY  is  the  most  prominent  feature  of 
the  subjects'  judgment  space,  and  considering  that  Accuracy  and  Reliabil- 
ity are  intertwined,  it  is  not  surprising  that  the  subjects'  primary 
distinction  between  reports  depends  on  the  source,  i.e.,  reliability  and 
accuracy  of  the  reports.  However,  it  is  somewhat  puzzling  to  find  no 
systematic  grouping  of  reports  on  the  basis  of  RELEVANCE  or  DIRECTNESS. 

The  structure  for  the  combined  subjects  shows  a three-factor  Bpace  which 
is  apparently  due  to  one  discrimination  on  the  basis  of  ACCURACY  and  a 
second  on  the  basi  of  RELEVANCE. 
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Means  and  Standard  Deviations  of  Overall  Qualities,  by  Reports 
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Table  10.  Report  Factor  Structure:  Trained  Subjects 


TRAINED  SUBJECTS 


Factor  I lb#Xt 
Message  Piacu 


Factor  II  (16%/ 


Factor  Coefficient 


factor  Coefficient 


46  Ini  Dir  reports  small  armi  Im  north  of 
Echamech.  alto  mortar  fee  received 


All  unit*  ?8th  Ini  Div  report  quiat  day  with 
scattered  small  emit  mortar  and  1 1 round*  ol 
artillery  lira. 


Ah  Force  reconnaissance  'areal*  enemy  tram* 
moving  south.  cart  on  udm#  and  activity  ei 
mar  dialling  yard*  .n  Trier 


112*i  Ini  Regt  Summary  unusual  enemy  harassing 
tmall  arm*  fire  during  hourt  ol  dar  kneit 


28th  Oi*  Arty  Reports  enemy  oatral  vicinity  CP 
228th  FA  Bn 


112th  Inf  Regt  leports  enemy  vehicles  heard 


1 10th  Int  Regt  reports  varied  enemy  traffic 
moving  to  u the  at  I 


109th  Int  Hegt  OF  personnel  report  sighting 
•tree  Tiger  Tank* 


1 10th  Inf  Regt  report*  German  patrol  on  tnendly 
tide  of  nver  recrotted  to  own  tide  river  twofien 
and  tenh 


Air  Force  bomber  ixlott  wetted  40  - truck  convoy 
moving  toudtaati  toward » Brttnjrq 


28ti  Oivition  Summary  enemy  urt>llery  kept  ir 
rear  area*,  lutle  fir  mg 


AM  ragts.  28th  Inf  Div  indicate  mcraaeed  early 
morning  tmall  arm*  fire,  otherwise  very  quiet 


106rh  tof  Di*  tummary  littenmg pottt  report 
unutual  motor  movementt.  three  enemy  pet  roll 
reported,  harassing  lues  h^it  new  enemy  be  tut -on 
reported 


1 10th  Inf  report!  German  petrol  on  friendly  tide 
of  nver  recrotted  to  own  ude  m rubber  boat 


1 12th  Inf  Regt  OF  per  tunnel  heard  three  enemy 


100th  Inf  Oi*  Summery  tmall  armi  fired  during 
darkneu  unutual  enemy  motor  activity,  eggretnve 
enemy  perroft 


1 12  th  Inf  Regt  OF  per  ton  net  heard  two  enemy 
truck!  before  dey'ifit 


1 10*i  Inf  Hegt  report*  hearing  a 
n 0746  0910 


4*«  Inf  Div  repur tt  enemy  vAicuier  activity 
l 12  Inf  Regt  during  r 


28th  Di*  Art  diary  Summery  much  a 
guardi  much  more  military  an*  warn h 
hrtt  time  overcoat*  note* 


more  confidant  an*  have  abrupt* 


106th  Inf  Di*  leport*  detar  ter  vayt  German  ISth 
Anny  moved  to  Cologne  and  Von  Rundtiedt  nat 
ordered  a withdraw * across  the  Rhine 


4th  Ini  Div  report*  enemy  deiariar  tayt  600  man 
r ecu  Mad  to  21201  VG  Div  to  taka  part  m forthconung 
attack  morale  good  to  very  good 


1 12th  Inf  Regt  reports  POW  itatat  Hq  26th  VG  Di* 
hat  moved  clot*  to  front  lines 


V Corpt  taper  tt  POW  tayt  Hq  362d  Art-llery  Regt 
probably  near  flettmgen 


4th  Inf  Div  IPW  Item  reports  German  civilian 
picked  up  > lutembourg  tie  tat  eiptouon  in 
mar  dialling  yard*  at  /werbrueckan  l-ed  up 
traffic  lor  pa  hours 


( fOth  fnf  Regt  report*  enemy  daterter  tayt  machine 
gun  letocated  to  cover  ruined  brid^  better 


4th  Inf  Div  Summery  POW't  report  three  divitioni 
•n  that,  ■ear  area  getting  ready  for  tag  attack 


28th  Inf  D-r  IPW  Team  report*  civilian!  state  two 
town*  acrotl  nver  are  full  o<  German  soldier  t. 
tome  mm# i being  laid 


VIII  Corps  Summary  POW  reports  1 16th  Ranter  Div 
moving  south  to  Trier 


?8th  Inf  Div  IPW  Team  reports  civilian  state*  German 
troop  concentrabont.  artillery  SS  troops,  and  nver 
crossing  equipment  moving  watt  from  Bitburg 


I09h  Inf  Regt  report!  deserter  tayt  hit  company 
it  neat  Bretdorl 
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The  three  report  factors  for  the  combined  subject  groups  (Table  11) 
account  for  78%  of  the  total  scale  variance  in  the  ratings  of  these 
reports.  Factor  III  accounts  for  more  of  this  total  variance  than  Factor 
II.  The  display  reverses  the  normal  ordering  of  factors  from  greater  to 
least  total  variance  to  indicate  the  order  of  the  unrotated  factors. 

This  is.  Factor  II  was  more  prominent  than  Factor  III  in  the  unrotated 
structure  but  accounted  for  less  variance  than  Factor  III  after  rotation. 

The  summarized  reports  shown  in  Table  11  reveal  that  the  three  factors 
have  relatively  clear  identifications.  Factor  I groups  those  reports 
containing  UNUSUAL,  LARGE  SCALE  or  THREATENING  ACTIVITY.  Factor  II  is 
largely  determined  by  DESERTER  and  CIVILIAN  REPORTS.  Factor  III  groups 
reports  of  PATROLS,  TRANSPORT  and  SMALL  ARMS  ACTIVITY.  As  a taxonomy  of 
these  reports,  the  structure  is  reasonably  coherent  and  significant. 

Supplementary  Analyses 

Several  additional  ways  of  examining  the  data  help  clarify  relation- 
ships among  the  rating  scales  and  among  the  reports  in  the  message  set. 
These  analyses  and  their  conclusions  are  discussed  briefly  here  and 
presented  in  more  detail  in  Appendixes  B-D. 

As  previously  established,  the  subjects  used  three  basic  factors  or 
concepts  in  judging  the  40  messages — ACCURACY,  RELEVANCE,  AND  DIRECTNESS. 
The  subjects'  approach  to  the  message  ratings  led  to  two  or  three  clusters 
of  related  messages,  messages  which  were  given  not  only  a similar  pattern 
of  ratings  but  which  had  definite  similarities  of  message  source  and 
content . 

Another  way  to  describe  relationships  among  the  messages  as  rated  by 
the  subjects  is  to  assign  factor  scores  to  the  messages  and  to  locate 
the  messages  in  the  three-dimensional  space  defined  by  the  rating  factors. 
The  factor  score  for  a given  report  was  obtained  by  taking  a weighted  sum 
of  all  the  ratings  on  that  report,  the  weights  being  proportional  to  the 
loadings  on  the  factor. 

Thus,  in  computing  the  ACCURACY  factor  score  for  a report,  the  ratings 
given  on  the  True-False  scale,  the  Reliable-Variable  scale,  etc.,  were 
given  high  weights.  Once  the  location  of  each  report  in  the  judgment 
space  had  been  determined,  it  was  possible  to  examine,  for  example,  the 
types  of  messages  identified  by  high  ACCURACY,  low  RELEVANCE,  and  low 
DIRECTNESS. 

A detailed  examination  of  the  report  factor  scores,  and  figures 
showing  the  geometric  relationships  among  reports,  is  presented  in 
Appendix  B.  The  ACCURACY  factor  clearly  played  a major  role  in  the 
subjects'  evaluation  of  the  reports;  the  RELEVANCE  and  DIRECTNESS  factors 
were  of  lesser  value. 
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This  result  was  not  surprising  considering  the  relative  percentages 
of  rating  variance  accounted  for  by  the  three  factors:  ACCURACY  is 
clearly  the  dominant  factor,  and  reports  would  be  expected  to  group 
principally  on  the  basis  of  their  ACCURACY  factor  scores.  However,  the 
composite  factor  scores  do  provide  meaningful  groupings  of  the  messages. 
If  single  rating  scales  are  developed  to  represent  these  factors,  ratings 
on  the  new  scales  may  be  valuable  for  structuring  or  organizing  data  to 
facilitate  its  use. 

Appendix  C presents  a more  detailed  analysis  of  the  ACCURACY  dimen- 
sion and  of  the  relationships  among  the  ratings  on  this  dimension  and  the 
traditional  Accuracy  and  Reliability  ratings. 

From  examination  of  the  inter-rating  correlation  in  Appendix  C,  it 
is  clear  that  the  Accuracy  and  Reliability  ratings  are  highly  related, 
despite  the  doctrinal  definition  of  these  ratings  as  independent. 

Apparently,  for  the  40  messages  used  in  the  experiment.  Accuracy  and 
Reliability  were  indeed  highly  related.  That  is,  all  accurate  reports 
might  have  come  from  reliable  sources,  and  subjects  mav  have  shown  good 
judgment  in  tying  the  two  closely  together. 

However,  examination  of  the  messages  (Appendix  A)  and  discussion  of 
some  messages  (Appendix  B)  show  instances  where  unreliable  sources 
produced  accurate  information  and  reliable  sources  provided  inaccurate 
information.  This  fact,  combined  with  the  data  in  Appendix  C,  strongly 
suggests  that  subjects  were  using  the  Reliability  rating  inappropriately 
by  tying  it  too  closely  to  their  judgment  of  Accuracy. 

In  addition,  it  appears  that  the  Global  Validity  ratings  may  provide 
a more  straightforward  representation  of  subjects'  estimates  of  informa- 
tion accuracy  than  the  standard  Accuracy  rating  does. 

The  multiple  regression  analyses  in  Appendix  D support  this 
conclusion  but  suggest  that  the  trained  subjects  indeed  may  have  been 
differentiating  between  Accuracy  and  Reliability.  Trained  subjects 
apparently  based  their  judgments  of  Accuracy  in  part  on  the  Reliability 
of  the  source,  but  they  did  not  seem  to  judge  Reliability  on  the  basis 
of  the  Accuracy  of  the  information  in  a given  message. 

The  supplementary  analyses,  then,  confirmed  the  finding  of  a 
dominant  concept  of  information  ACCURACY.  However,  relationships  of  the 
standard  ratings  of  Accuracy  and  Reliability  to  this  concept  and  to  each 
other  remain  somewhat  unclear.  It  is  apparent  that  these  standard 
ratings  do  not  capture  the  full  meaning  of  the  ACCURACY  dimension  as 
derived  from  the  subject  data.  To  explore  more  fully  the  interrelation- 
ships among  the  various  ratings,  the  following  steps  need  to  be  taken: 

1.  Replicate  the  present  experiment  using  a different  sample  of 
analysts  with  more  extensive  military  experience  and  Intelligence 
training. 
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2.  Replicate  the  present  experiment  using  a different  sample  of 
messages . 

3.  Develop  rating  scales  specifically  designed  to  capture  the 
meaning  of  the  principal  judgment  factors  identified  in  the  present 
experiment . 

The  data  from  these  experiments  should  verify  the  stability  and 
generality  of  judgment  factors  appropriate  for  the  evaluation  of  combat 
intelligence  data.  In  addition,  some  insight  would  be  obtained  about  the 
potential  usefulness  of  new  rating  scales  specifically  designed  to  tap 
these  factors. 


SUMMARY  AND  CONCLUSIONS 

The  present  experiment  was  an  exploration  of  the  structure  of  combat 
intelligence  data  as  seen  through  the  eyes  of  intelligence  analysts.  A 
selection  of  40  messages  was  used  as  representative  of  intelligence 
information  received  at  division  level. 

Ratings  were  requested  from  34  enlisted  personnel  just  beginning  an 
intelligence  analyst  course  at  the  U.S.  Army  Intelligence  Center  and 
School.  An  additional  21  enlisted  personnel,  just  finishing  the  course, 
were  asked  for  similar  ratings.  Members  of  both  groups  were  relatively 
inexperienced  in  intelligence  analysis.  Attributes  of  the  reports  to  be 
rated  included  specific  qualities  of  content  as  well  as  the  traditional 
Accuracy  and  Reliability  ratings  and  ratings  of  the  overall  validity  or 
credibility  of  the  report. 

The  purpose  was  twofold,  (1)  to  ascertain  the  underlying  conceptual 
structure  employed  by  the  subjects  in  evaluating  typical  messages 
containing  intelligence  information,  and  (2)  to  assess  the  role  of  the 
traditionally  employed  reliability  and  accuracy  ratings  prescribed  for 
the  intelligence  analyst. 

With  respect  to  the  conceptual  structure,  it  was  found  that  analysts 
had  an  implicit  judgmental  structure  within  which  they  evaluated  the 
attributes,  qualities,  or  characteristics  of  the  reports  upon  which  they 
were  called  to  pass  judgment. 

The  dimensions  of  this  judgmental  space,  determined  by  multivariate 
analyses,  were  Accuracy,  Relevance,  and  Directness.  In  combination, 
these  independent  dimensions  accounted  for  more  than  82%  of  the  total 
variance  of  the  combined  evaluations  of  the  report  sample  as  made  by  the 
55  analysts  (Accuracy,  57%;  Relevance,  19%;  Directness,  6%). 
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With  regard  to  the  traditionally  used  Accuracy  and  Reliability  scales, 
the  evidence  of  prior  studies  was  verified  that  the  scales  are,  in  fact, 
not  independent.  Findings  also  indicated  that  the  judgments  of  Accuracy 
as  traditionally  measured  were  significantly  contaminated  by  apparently 
unfounded  judgments  of  source  reliability,  particularly  on  the  part  of 
the  trained  analysts. 

For  the  untrained  subjects,  however,  the  two  measurements  were 
completely  interchanged.  In  contrast,  a specially  devised  Global  scale 
of  report  validity  was  found  less  contaminated  and  to  have  generally 
better  measurement  validity  for  report  Accuracy. 

These  findings  imply  that  scales  for  evaluating  quality  of  intelli- 
gence data  could  be  derived  to  provide  a more  effective  vehicle  for 
rating  of  data  quality  and  transmission  of  ratings. 

Scales  more  closely  paralleling  the  analysts'  conceptual  framework 
would  permit  a more  natural  judgment  process,  and  the  ratings  given 
would  more  accurately  reflect  the  rater's  feelings  about  the  data  being 
rated.  The  use  of  0-100  rather  than  1-6  scales  might  also  contribute  to 
more  effective  communication  between  the  person  rating  the  information 
and  the  analyst  or  decisionmaker  utilizing  the  information. 

Before  development  of  new  scales  and  procedures,  however,  the  results 
obtained  in  the  present  experiment  need  to  be  validated  by  (1)  replica- 
ting the  study  using  a sample  of  trained  intelligence  officers  with 
combat  experience  and  (2)  repeating  the  study  using  a different  message 
set  and  associated  scenario. 
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APPENDIX  A 

MESSAGES  RATED  BY  SUBJECTS 


BATTLE  of  the  BULGE 
28th  Infantry  Division 
Selected  Messages  for  10-15  December  1944 

► ’ 

' 

[ 

Alignment  of  Units  in  VIII  Corps 
from  North  to  South 


106  Infantry  Division  with  4th  Cavalry 
Regiment  attached  on  North  Flank 


JRth  Infantry  Division 
11 'th  Infantry  Regiment 
110th  Infantry  Regiment 
109 th  Infantry  Regiment 

9th  Armored  Division  (60th  Armored  Infantry) 
Battalion  of  Combat  Command  A on  line 


4th  Infantry  Division 
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Message  Nr  1 

10  December  1944  28th  Infantry  Division 

1000  hours 

The  3d  Battalion,  109th  Infantry  Regiment  reports  that  a deserter 
captured  in  the  vicinity  of  coordinates  9241  at  0540  hours  is  from  the 
1st  Battalion,  915th  Regiment,  352d  Volks grenadier  Division.  The 
prisoner  of  war  stated  his  company  is  located  in  the  vicinitv  of  BIESDORF 
(9743). 


Message  Nr  2 

10  December  1944  28th  Infantry  Division 

1500  hours 

3d  Battalion,  110th  Infantry  Regiment  reports  it  observed  an  8-man 
enemy  patrol  in  the  vicinity  of  coordinates  851581  at  1115  hours.  Enemy 
withdrew  eastward  and  crossed  river  in  a rubber  boat  which  was  found  in 
the  vicinity  of  854584. 


Message  Nr  3 

10  December  1944  28th  Infantry  Division 

1200  hours 

3d  Battalion,  110th  Infantry  Regiment  reports  that  between  0900-1130 
hour 8 it  sighted  15  horse-drawn  vehicles  and  1 staff  car  between  coordi- 
nates 890570  and  897562. 


Message  Nr  1 

11  December  1944  28th  Infantry  Division 

1500  hours 

3d  Battalion,  110th  Infantry  Regiment  reports  seeing  a 6-man  German 
patrol  walking  towards  the  Our  River  in  the  vicinity  of  coordinates 
871541  at  1100  hours.  Observers  did  not  fire,  but  followed  the  enemy 
patrol.  Tracks  led  to  the  river  in  the  vicinity  of  coordinates  875543. 
They  could  not  determine  whether  the  enemy  patrol  used  a boat  to  re-cross 
the  river  or  waded.  They  kept  watch  in  the  area  until  1300  hours,  but 
did  not  see  any  more  activity.  The  current  in  the  river  was  swift  and 
the  water  was  level  with  the  banks . Observers  returned  to  their  unit  at 
1400  hours . 
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Message  Nr  2 

11  December  1944  28th  Infantry  Division 

1200  hours 

28th  Infantry  Division  Interrogation  of  Prisoner  of  War  Team  reports 
reliable  civilians  state  that  as  of  10  December: 

1.  NIEDERGECKLER  (9753)  and  GEICHLINGEN  (9451)  were  full  of 
German  soldiers. 

2.  Many  mines  were  being  laid  in  the  vicinity  of  BAULER  (9052). 

Message  Nr  3 

11  December  1944  28th  Infantry  Division 

1200  hours 

106th  Infantry  Division  reports  a deserter  who  surrendered  9 December 
says  he  is  from  Headquarters  15th  Army  which  recently  moved  from  the 
Rotterdam  area  to  a location  in  the  Cologne-Bonn-Dueren  area.  Rumor  at 
his  Headquarters  is  that  Field  Marshal  von  Rundstedt  has  ordered 
withdrawal  to  the  East  bank  of  the  Rhine  because  shortage  of  reserves 
will  make  defense  west  of  the  Rhine  no  longer  feasible  after  the  first 
of  the  year. 


Message  Nr  4 

11  December  1944  28th  Infantry  Division 

2400  hours 

4th  Infantry  Division  reports  enemy  rifle  and  pistol  fire  in  northern 
part  of  sector  (north  of  ECHTERNACH)  at  1400  hours.  About  50  rounds 
each  of  80  mm  and  120  mm  mortar  fire  were  received  during  the  day. 

Message  Nr  5 

11  December  1944  28th  Infantry  Division 

2400  hours 

Summary  of  Enemy  activity  along  28th  Infantry  Division  front  for  11 
December . 

1.  All  units  reported  a quiet  day. 

2.  Scattered  rifle,  pistol  and  mortar  fire  were  received  all  along 
the  front. 

3.  11  rounds  of  light  artillery  fire  were  also  received. 
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Message  Nr  1 

12  December  1944  28th  Infantry  Division 

1000  hours 

3d  Battalion,  109th  Infantry  Regiment  reports  observation  post 
personnel  sighted  three (3)  TIGER  Tanks  in  the  vicinity  of  Road  Junction 
at  coordinates  952458  south  of  NIEDERSGEGEN  at  0905  hours. 

Message  Nr  2 

12  December  1944  28th  Infantry  Division 

1200  hours 

110th  Infantry  Regiment  reports. 

1.  Four (4)  horse-drawn  vehicles  observed  moving  southeast  in  the 
vicinity  of  88875  at  1019  hours. 

2.  0ne(l)  half-track*,  several  horse-drawn  vehicles  carrying  troops 
and  one(l)  horse-drawn  artillery  piece  (weapon)  observed  moving  south- 
west in  the  vicinity  930602  at  1100  hours. 

*Vehicle  with  wheels  in  front  and  tank-like  tracks  in  back. 

Message  Nr  3 

12  December  1944  28th  Infantry  Division 

1400  hours 

VIII  Corps  G2  Air  reports  pilots  of  heavy  bombers  returning  from  mission 
sighted  a 40-truck  convoy  moving  southeast  from  WAXWEILER  (0167)  to 
BITBURG  (1454)  at  1100  hours. 


Message  Nr  4 

12  December  1944  28th  Infantry  Division 

2400  hours 

112th  Infantry  Regiment  reports: 

1.  Occasional  rifle  and  pistol  fire  and  flares  all  along  front 
during  hours  of  darkness . 

2.  Sound  of  several  vehicles  heard  in  the  vicinity  of  R0SCHEID 
(8970)  at  1950  hours. 

3.  Single  enemy  planes  flew  over  regimental  area  at  2015  and  2113 
hours . 
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Message  Nr  5 

12  December  1944 

2000  hours 

VIII  Corps  relayed  following  message  from  First  Army:  "Train  movements 
indicate  that  the  buildup  of  enemy  forces  on  the  western  slope  of  the 
MOSELLE  Valley  continues.  The  Gross  Deutschland  Division*  has  again 
been  reported  in  the  area  by  a prisoner  of  war.  A conservative  estimate 
would  place  at  least  two(2)  Volks grenadier  Divisions  and  one(l)  Panzer 
Division  in  the  enemy’s  rear  area  opposite  VIII  Corps.  These  units 
maintaining  have  not  used  their  radios." 

*A  Panzer  (armored)  division. 

Message  Nr  1 

13  December  1944  28th  Infantry  Division 

0400  hours 

28th  Division  Artillery  reports  that  at  0300  hours  the  229th  Field 
Artillery  Battalion  detected  a two (2)  man  enemy  patrol  in  the  vicinity 
of  its  CP.  The  patrol  withdrew  when  fired  on. 

Message  Nr  2 

13  December  1944  28th  Infantry  Division 

1550  hours 

3d  Battalion,  110th  Infantry  Regiment  reports  that  Patrol  Nr.  6 saw 
little  activity  in  the  vicinity  of  ROTH  (915482)  at  2000  hours  on  12 
December. 

Message  Nr  3 

13  December  1944  28th  Infantry  Division 

2100  hours 

4th  Infantry  Division  reports  that  a deserter  from  the  2d  Battalion, 
316th  Regiment,  212th  Volksgrenadier  Division  stated: 

a.  The  230th  Regiment  of  the  Division  was  on  line  for  about  two 
weeks  but  is  now  in  reserve. 
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b.  Approximately  600  men  on  special  types  of  duty  in  rear  areas  were 
called  back  to  the  316th  and  423rd  regiments  of  the  212th  Volksgrenadier 
Division  by  a secret  order  on  11  December.  They  were  told  to  return  to 
their  regiments  immediately  because  they  were  needed  for  an  upcoming 
attack  against  the  thinly  held  American  line.  These  men  for  the  most 
part  had  been  attending  noncommissioned  officer  schools. 

c.  Morale  of  German  Troops  in  his  division  is  good  to  very  good. 

Message  Nr  4 

13  December  1944  28th  Infantry  Division 

2400  hours 

112th  Infantry  Regiment  reports  that  troops  along  the  front  heard  sounds 
of  enemy  vehicles  during  the  evening. 

1.  Tracked  vehicles  in  the  vicinity  of  coordinates  9272  at  2130 
hours . 

2.  Trucks,  motorcycles  and  possibly  tanks  in  town  of  R0SCHEID(89 70) 
at  2200  hours. 

3.  Motors  in  the  vicinity  of  coordinates  874693  from  2215-2245  hours. 

Message  Nr  5 

13  December  1944  28th  Infantry  Division 

2400  hours 

106th  Infantry  Division  Summary  stated: 

1.  Indiscriminate  rifle  and  pistol  firing  along  entire  front  during 
hours  of  darkness. 

2.  Unusual  motor  activity  in  the  vicinity  of  SCHEID  (064963)  and 
BERG (908750)  in  the  past  24  hours  suggests  presence  of  new  troops  in  the 
area. 

3.  Aggressive  enemy  patrols  displayed  increased  activity  along 
entire  front  during  past  24  hours. 

Message  Nr  6 

13  December  1974  28th  Infantry  Division 

2400  hours 

VIII  Corps  Intelligence  Summary  reports  that  a message  forwarded  from 
First  Army  at  2000  hours  states  that  a prisoner  of  war  says  the  116th 
Panzer  Division  has  moved  south  to  the  Trier (2129)  area.  Its  rear 
echelon  including  division  headquarters  left  MUENCHEN-GLADBACK  on  6 
December.  First  Army  notes  this  report  seems  to  confirm  a southward 
movement  earlier  reported  on  the  116th  Panzer  Division. 
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14  December  1944 
0600  hours 


Message  Nr  1 

28th  Infantry  Division 


112th  Infantry  Regiment  reports  that  observation  post  personnel  heard 
two  (2)  trucks  on  Hill  at  coordinates  878698*.  Time  sounds  were  heard 
was  0405  hours. 

*SE  of  ROSCHEID  (8970)  and  opposite  the  boundary  of  the  2d  and  3d 
Battalion  112th  Infantry. 


Message  Nr  2 

14  December  1944  28th  Infantry  Division 

0630  hours 

2d  Battalion,  112th  Infantry  Regiment  reports  wounded  POW  captured  near 
LIELER  (838705)  at  132345  hours  while  on  patrol  states  he  heard  the 
headquarters  of  the  26th  Volksgrenadier  Division  moved  up  to  ESCHFELD 
(90269 2) -ROSCHEID  (8970)  area  on  5 December. 

Message  Nr  3 

14  December  1944  28th  Infantry  Division 

1000  hours 

110th  Infantry  Regiment  reports  that  vehicular  movements  have  been  heard 
in  the  -vicinity  of  coordinates  854680  at  0745,  0830,  and  0910  hours. 
Direction  and  volume  undetermined. 


Message  Nr  4 

14  December  1944  28th  Infantry  Division 

1015  hours 

1st  Battalion,  110th  Infantry  Regiment  reports  that  an  enemy  deserter 
picked  up  at  132300  hours  in  TINTESMUEHLE,  a small  village  at  coordinates 
850673  at  the  bend  in  the  river  East  of  HEINERSCHEID  (821675)  and  SE  of 
KALBORN  (840682)  says  he  was  born  in  STRASBOURG.*  He  stated  that  the 
machine  gun  covering  the  ruins  of  the  bridge  at  coordinates  850672  was 
moved  on  7 December  to  a better  location  100  meters  from  its  former 
position  where  it  can  fire  directly  down  the  center  line  of  the  old 
bridge . 

*0n  west  back  of  the  Rhine,  Southeast  of  the  VIII  Corps  sector. 
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Message  Nr  5 

14  December  1944  28th  Infantry  Division 

1400  hours 

112th  Infantry  Regiment  reports  that  vehicular  movements  were  heard  again 
behind  hill  at  coordinates  878698  and  vicinity  873687  at  1300  hours. 

Message  Nr  6 

14  December  1944  28th  Infantry  Division 

1200  hours 

3d  Battalion,  110th  Infantry  Regiment  reported  that  a large  formation  of 
Enemy  Infantry  was  heard  in  GEMUEND  (871557)  at  102200  December. 

Suspects  assembly  area  for  raid  into  110th  sector.  Requests  Corps 
Artillery  bring  all  possible  fire  on  the  area  as  soon  as  possible. 

Message  Nr  7 

14  December  1944  28th  Infantry  Division 

1600  hours 

The  28th  Division  Interrogation  of  Prisoner  of  War  Team  reported  that  a 
woman  who  crossed  into  Germany  on  10  December  was  arrested  and  taken  to 
BITBURG(1353) . She  escaped  and  walked  back  into  Luxembourg  where  she 
reported  to  police  this  morning.  She  is  vouched  for  by  the  Luxembourg 
police  as  thoroughly  reliable.  Her  statement  follows:  "I  saw  many 
horsedrawn  vehicles,  pontoons,  small  boats,  and  other  river-crossing 
equipment  coming  from  the  direction  of  BITBURG  and  moving  west  through 
GEICHLINGEN  (9451).  In  BITBURG,  I overhead  some  military  personnel 
saying  it  had  taken  three  weeks  to  get  there  from  Italy.  There  were  also 
some  troops  in  the  town  with  gray  uniforms  with  black  collar  patches.* 

I also  saw  many  artillery  pieces,  both  horse-drawn  and  carried  on 
trucks." 

*Uniform  of  SS (Elite  Nazi)  troops. 

Message  Nr  8 

14  December  1944  28th  Infantry  Division 

1930  hours 
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VIII  Corps  G2  Air  reported  that  an  Air  Force  visual  reconnaissance 
mission  this  afternoon  made  the  following  observations: 

1.  Train  consisting  of  50  small  freight  cars  was  moving  south  in 
the  vicinity  of  coordinates  2050. 

2.  20  more  small  freight  cars  were  on  siding  in  the  same  general 
area. 

3.  Considerable  activity  in  the  marshalling  yard  at  TRIER(2129)  . 


14  December  1944 
2000  hours 


Message  Nr  9 

28th  Infantry  Division 


112th  Infantry  Regiment  reports  that  observation  post  personnel  heard 
three (3)  vehicles  headed  west  at  1840  hours.  It  sounded  as  if  they 
were  pulling  heavy  loads  up  a hi.1!  in  the  vicinity  of  coordinates  880696. 

Message  Nr  10 

14  December  1944  28th  Infantry  Division 

2130  hours 


3d  Battalion,  110th  Infantry  Regiment  reports  that  a patrol  found  a dead 
German  in  the  vicinity  of  coordinates  847594  at  2000  hours.  Only  item 
of  military  value  found  was  a map  stamped  2d  Company,  78th  Infantry 
Regiment.*  The  map  was  DASBURG,  Sheet  5902.  Man  appeared  to  have  died 
about  24  hours  ago  from  a bullet  wound. 

*This  would  indicate  the  map  came  from  the  1st  Battalion,  78th  Infantry 
Regiment,  26th  Volksgrenadier  Division. 


Message  Nr  11 

14  December  1944  28th  Infantry  Division 

2400  hours 

112th  Infantry  Summary  stated  that  there  was  unusual  intermittent 
harassing  small  arms  fire  from  140001  hours  until  dawn.  It  started 
again  at  1810  hours.  At  2000  hours  there  was  a marked  increase  in 
volume.  At  least  one  burst  of  automatic  pistol  or  machine  gun  fire 
was  reported  at  different  points  along  the  front  and  extending  to  the 
south  of  the  112th  sector,  at  an  average  rate  of  one  burst  every  five 
minutes.  This  activity  was  still  continuing  at  142400  hours. 
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Message  Nr  12 

14  December  1944  28th  Infantry  Division 

0600  hours 

V Corps  G2  Periodic  Report  for  13  December  had  a significant  item  under 
"Enemy  Dispositions."  It  stated  that  a prisoner  of  war  from  the  2d 
Company,  942d  Infantry  Regiment,  353d  Volksgrenadier  Division  who  was 
captured  in  the  V Corps  sector  reported  that  the  former  Command  Post 
of  the  353d  Artillery  Regiment  while  opposite  the  VIII  Corps  was  in  the 
vicinity  of  BETTINGEN  (0450) . He  stated  that  it  is  customary  for 
relieving  units  to  occupy  the  same  positions  as  those  of  the  relieved 
unit.  This  points  to  the  possibility  that  the  same  Command  Post  is  now 
occupied  by  the  headquarter  of  the  352d  Artillery  Regiment. 

Message  Nr  13 

14  December  1944  28th  Infantry  Division 

2400  hours 

28th  Division  Artillery  Summary: 

1.  During  the  past  week  the  enemy  has  kept  his  artillery  well  back 
from  the  front . 

2.  No  enemy  firing  has  been  reported  at  night. 

3.  No  counterbattery*  fire  has  been  reported. 

4.  Enemy  artillery  activity  for  14  December: 

a.  11  rounds,  reportedly  all  75mm,  were  fired  into  the 
sector  of  the  112th  Infantry  Regiment. 

b.  Approximately  six(6)  rocket  projectiles  believed  fired  from 
near  coordinates  950440  heard  and  seen  at  1940  hours . 

*Firing  at  our  artillery 

Message  Nr  14 

14  December  1944  28th  Infantry  Division 

2400  hours 

106th  Infantry  Division  Summary: 

1.  Listening  post  personnel  reported  heavy  Increase  in  motor 
vehicle  movement  particularly  vie  ORMONT(0892) , SCHEID(0696) , and 
BRANDSCHEID(9881)  before  dawn  and  after  dark.  They  heard  sounds 
suggesting  unloading  of  troops. 

2.  Harassing  fires  very  light. 

3.  At  least  three(3)  Enemy  patrols  reported  in  106th  sector. 
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4.  Two(2)  Prisoner  of  War,  from  3d  Company,  295th  Regiment,  18th 
Volks grenadier  Division  report  1st  Battalion,  295th  Regiment  in  the 
ORMONT  area. 


Message  Nr  15 

14  December  1944  28th  Infantry  Division 

2400  hours 

4th  Infantry  Division  Summary  stated  that  vehicular  traffic  was  reported 
on  the  12th  Infantry  Regiment's  front  during  the  evening.  Otherwise  it 
was  a quiet  day. 

(Note:  12th  Infantry  occupies  the  northern  regimental  sector  of  the  4th 
Infantry  Division  from  ECHTERNACH  north  to  the  9th  Armored  Division's 
sector. ) 


Message  Nr  1 

15  December  1944  28th  Infantry  Division 

0600  hours 

4th  Infantry  Division  Interrogation  of  Prisoner  of  War  Team  report  states 
that  a German  national  from  SAARBRUECKEN*  was  picked  up  while  visiting 
relatives  vicinity  Luxembourg  City  on  6 December.  He  claims  he  is  an 
employee  of  the  German  Railroad  System.  He  further  states  that  on 
30  November  the  marshalling  yards  in  ZWEIBRUECKEN*  were  tied  up  for  six 
hours  when  two  carloads  of  explosives  collided  and  exploded. 

*0pposite  Third  Army  zone  to  the  southeast. 

Message  Nr  2 

15  December  1944  28th  Infantry  Division 

2400  hours 

109th,  110th,  and  112th  Infantry  Regiments  Summaries  all  indicate  marked 
Increase  of  small  arms  harassing  fire  between  0400  and  0800  hours, 
followed  by  a very  quiet  afternoon  and  evening. 
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Message  Nr  3 

28th  Infantry  Division 


2400  hours 

28th  Division  Artillery  Summary: 

1.  Observation  Post  101  (964414)  observers  reported  at  1040  hours 
that  enemy  in  front  of  them  were  equipped  with  overcoats  for  the  first 
time . 

2.  During  the  day  when  guards  were  changed,  enemy  soldiers  double- 
timed  to  and  from  their  posts  and  did  quite  a bit  of  saluting  not 
observed  before. 

3.  Vicinity  963436  and  958435  enemy  did  a lot  of  moving  to  and  from 
pillboxes . 

4.  There  was  also  quite  a bit  of  foot  traffic  vicinity  965439  all 

day. 


Message  Nr  4 

15  December  1944  28th  Infantry  Division 

2400  hours 

4th  Infantry  Division  Summary  states  that  two (2)  POW  from  the  44th 
Fortress  Machine  Gun  Battalion  were  captured  at  995196  at  1830  hours. 
They  said  they  were  told  two  days  ago  that  a Panzer  Division  and  two(2) 
Infantry  Divisions  were  in  the  rear  of  their  area  and  that  those  units 
would  soon  launch  an  offensive  of  major  proportions. 

Message  Nr  5 

15  December  1944  28th  Infantry  Division 

2400  hours 

9th  Armored  Division  Summary  states: 

1.  Enemy  outpost  personnel  were  more  in  evidence  than  usual  during 
the  day  and  appeared  more  confident. 

2.  Abrupt  change  of  routine  of  personnel  on  the  other  side  of  the 
river  strongly  suggests  new  troops  may  have  arrived  on  the  front. 
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Message  Nr  6 

15  December  1944  28th  Infantry  Division 

2400  hours 

106th  Infantry  Division  Summary: 

1.  Considerable  vehicular  activity  was  heard  in  the  vicinity  of 
coordinates  996806  at  0045  hours  and  all  along  the  front  from  1800 
hours  to  the  time  of  this  report. 

2.  An  enemy  patrol  consisting  of  34  men  was  reported  in  ALLMUTHUN 
(040948)  during  the  early  morning. 

3.  Enemy  observation  planes  were  active  over  throughout  the  divi- 
sion sector  early  this  evening. 

4.  A prisoner  of  war  from  the  2d  Battalion  295th  Regiment,  18th 
Volksgrenadier  Division  who  was  captured  this  afternoon  says  his  unit 
arrived  in  this  area  on  the  afternoon  of  14  December  relieving  an 
unknown  unit  which  was  to  be  pulled  out  for  an  attack  along  the  entire 
front  between  the  17th  and  Christmas.  P0W  had  heard  that  SS*  troops 
were  to  be  used  and  that  they  would  employ  infiltration  tactics.  He 
further  stated  that  in  the  area  of  his  unit  there  were  large  generators 
and  40  large  searchlight  reflectors. 

*Elite  Nazi 
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APPENDIX  B 

REPORT  FACTOR  SCORES  OF  THE  SCALE  STRUCTURE 

One  way  to  depict  the  taxonomy  resulting  from  the  scale  structure 
obtained,  as  applied  to  these  reports,  is  to  display  the  factor  scores 
of  the  40  reports  in  the  calculated  judgmental  space. 

Such  displays  have  been  produced  by  creating  three  dimensional 
models  (Figures  B1  through  B3) . For  each  model,  the  three  axes  of 
ACCURACY,  RELEVANCE  and  DIRECTNESS  form  a coordinate  system  for 
displaying  the  location  of  each  report  in  the  subjects'  judgment  space. 

Locations  of  each  report  in  this  factor  space  are  based  on  the  linear 
equation  of  the  weighted  combinations  of  each  of  the  scale  ratings,  i.e., 
the  factor  scores  of  the  reports  calculated  from  their  weighted  scale 
averages.  For  the  purposes  of  the  display,  the  factor  scores  have  been 
scaled  relative  to  their  maxima  in  each  dimension. 

Although  this  produces  some  distortion,  the  distortion  is  constant 
in  all  dimensions.  Thus,  relativized,  the  report  locations  are  merely 
spread  out  more  than  they  would  be  if  not  so  scaled,  but  their  positions 
relative  to  each  other  are  as  found  in  the  analyses. 

For  purposes  of  the  following  discussion,  the  three  dimensional 
(continuous)  structure  of  the  scale  factor  space  for  combined  subjects 
is  cut  into  eight  discrete  compartments  (or  octants) . The  reports 
falling  into  each  of  the  compartments  of  the  space  are  then  analyzed. 

The  three  factorial  dimensions  of  the  judgmental  space  depicted  by 
Figure  B3  for  combined  subjects  can  be  envisioned  as  shown  in  Figure  B4. 
In  Figure  B4a,  for  example,  the  three  dimensions  are  shown  as  inter- 
secting axes  with  each  dimension  extending  from  the  origin  or  0-point 
one  unit  measurement  in  both  positive  and  negative  directions.  The 
Accuracy  and  Relevance  axes  define  a horizontal  plane.  Accuracy  and 
Directness  dimensions  define  a vertical  plane  as  do  the  Relevance  and 
Directness  axes.  If  each  of  the  dimensions  of  the  structure  is  consid- 
ered to  have  a positive  (+)  and  a negative  (-)  end  or  pole,  the  octants 
formed  by  the  various  combinations  of  the  poles  of  the  three  dimensions 
can  be  defined,  in  order,  by  the 

+ + + poles  of  the  Accuracy,  Relevance,  and  Directness  factors 

(Octant  1),  by  the  + + - (Octant  2),  + - + (Octant  3) , + - - 

(Octant  4),  - + + (Octant  5),  - + - (Octant  6),  - - + (Octant  7) 

and  - - - (Octant  8).  The  order  of  poles  is  Accuracy,  Relevance, 

and  Directness. 

The  three  planes  intersect  along  lines  passing  through  the  point  of 
origin  of  the  axes.  Octant  1 is  formed  by  the  positive  axes  of  all 
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three  dimensions,  as  shown  in  Figure  BAb , i.e.,  it  has  cubic  form  and 
lies  in  the  upper  far  right  hand  corner  of  the  judgmental  space. 

Octant  2 lies  directly  below  Octant  1 and  forms  the  lower  far  right 
hand  compartment  of  the  judgmental  space,  as  shown  in  Figure  BAc.  In 
a similar  manner.  Octants  3-8  may  be  envisioned  from  their  polar 
definitions.  Figure  BAd  shows  Figure  B3  subdivided  into  octants  but 
without  the  report  positions  being  depicted. 

There  is  high  face  (content)  validity  for  the  classification  of  each 
of  the  reports  contained  within  each  octant  as  described  below.  The 
reports  falling  within  each  octant  are  enumerated  and  their  collective 
identity  summarized.  In  addition,  a typical  report  is  cited  to  provide  an 
appreciation  for  the  type  of  report  falling  within  that  part  of  the 
judgmental  space  identified  by  the  octant  specified. 

Octant  1:  + Accuracy  + Relevance  + Directness.  Reports  10,  18,  19,  2A 

26,  28  and  AO  have  values  greater  than  zero  on  each  of  the  three  scalar 
factors.  The  common  characteristic  of  these  reports  is  enemv  activity 
seen  or  heard  by  U.  S.  military  forces,  local  friendly  observers,  and 
POWs. 

The  reports  came  from  subordinate  units  of  the  28th  Infantry 
Division,  including  the  Interrogation  of  Prisoner  of  War  Team;  adjacent 
divisions,  and  from  local  police  units  friendly  to  the  Allied  cause. 

The  activity  consisted  of  movements  of  supplies,  troops,  artillery,  and 
river  crossing  equipment  by  trucks  and  horse-drawn  vehicles;  small  arms 
firing  and  enemy  reconnaissance  in  the  form  of  patrols  and  observation 
aircraft. 

The  troop  movements  varied  in  size  from  patrols  of  small  numbers  of 
men  to  the  entry  into  the  area  opposite  the  VUIth  Corps  of  a whole 
Panzer  (Tank)  division  and  the  appearance  of  elite  SS  Troops . POW 
reports  indicated  preparations  for  an  enemy  attack,  and  those  of 
friendly  units  described  enemy  actions  as  unusual  and  aggressive. 

Report  AO  (coordinate  values  of  .17,  .50  and  .21)  is  among  the 
highest  rated  reports  in  this  octant.  The  text  of  the  report  was  as 
follows : 

15  December  19AA  28th  Infantry  Division 

2A  hours 

106th  Infantry  Division  Summary: 

1.  Considerable  vehicular  activity  was  heard  in  the  vicinity  of 
coordinates  996806  at  00A5  hours  and  all  along  the  front  from  1800 
hours  to  the  time  of  this  report. 

2.  An  enemy  patrol  consisting  of  3A  men  was  reported  in  ALLMITTHUN 
(0A09A8)  during  the  early  morning. 


B-2 


ART  TP  286 


3.  Enemy  observation  planes  were  active  overhead  throughout  the 
division  sector  early  this  evening. 

4.  A prisoner  of  war  from  the  2d  Battalion  295th  Regiment,  18th 
Volks grenadier  Division  who  was  captured  this  afternoon  says  his  unit 
arrived  in  this  area  on  the  afternoon  of  14  December  relieving  an  unknown 
unit  which  was  to  be  pulled  out  for  an  attack  along  the  entire  front 
between  the  17th  and  Christmas.  POW  had  heard  that  SS  troops  were  to  be 
used  and  that  they  would  employ  infiltration  tactics.  He  further  stated 
that  in  the  area  of  his  unit  there  were  large  generators  and  40  large 
searchlight  reflectors. 

Octant  2:  + Accuracy  + Relevance  - Directness.  Reports  11,  17,  27, 

30,  33,  37  and  39  are  located  in  this  octant. 

The  common  element  of  these  reports  is  enemy  changes  all  along  the 
FEBA  of  the  VII  Corps  from  the  106th  Infantry  Division  on  the  North  of 
the  28th  Infantry  Division  through  the  28th  and  continuing  into  the  9th 
Armored  Division  on  the  South  of  the  28th,  together  with  sizeable  truck 
convoy  and  train  movements  carrying  supplies  and  possible  troops  from 
the  North  into  the  enemy  rear  areas  opposite  the  VIII  Corps. 

Four  of  the  seven  reports  in  this  octant  came  from  units  other  than 
the  28th  Infantry  Division.  There  was  one  each  from  the  106th  Division 
and  9th  Armored  Division  and  two  from  VIII  Corps  G2 . 

Specifics  other  than  truck  convoy  and  train  movements  included 
increased  small  arms  fire  during  hours  of  darkness  and  increased  motor 
vehicle  movements,  both  opposite  the  northern  part  of  the  VIII  Corps 
sector,  and  obvious  changes  in  enemy  front  line  units  such  as  increased 
activity,  better  discipline,  better  clothing  for  winter  weather,  greater 
self-confidence,  and  a different  schedule  of  daily  operations. 

The  text  of  Report  37,  as  representative  of  the  octant,  was  as 
follows: 

15  December  1944  28th  Infantry  Division 

2400  Hours 

28th  Division  Artillery  Summary: 

1.  Observation  Post  101  (964414)  observers  reported  at  1040  hours 
the  enemy  in  front  of  them  were  equipped  with  overcoats  for  the  first 
time. 


2.  During  the  day  when  guards  were  changed,  enemy  soldiers  double- 
timed  to  and  from  their  posts  and  did  quite  a bit  of  saluting  not 
observed  before. 

3.  Vicinity  936436  and  958435  enemy  did  a lot  of  moving  to  and  from 
pillboxes . 
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4.  There  was  also  quite  a bit  of  foot  traffic  vicinity  965439  all 

day. 

Octant  3:  + Accuracy  - Relevance  + Directness.  Reports  2,  7,  9,  and  20 

are  in  this  octant.  These  reports  in  the  3d  Octant  are  characterized  by 
friendly  front  line  troops  seeing  or  hearing  minor  enemy  activity  such 
as  small  enemy  patrols,  small  numbers  of  tanks  and  trucks  and  small 
arms,  mortar  and  artillery  fire. 

Most  analysts  apparently  failed  to  note  that  Report  9 is  a false 
report,  it  being  impossible  to  see  the  location  at  which  the  tanks  were 
reported  from  the  observation  post  whose  personnel  made  the  report. 

In  addition,  the  analysts  failed  to  appreciate  the  fact  that  the 
appearance  of  TIGER  tanks  close  to  the  front  was  unusual,  and  if  they 
believed  the  report  true  they  should  have  perceived  a relevant  threat 
since  the  opposing  Volksgrenadier  Divisions  were  not  normally  equipped 
with  any  tanks,  much  less  the  heavy  TIGERS. 

Report  2 is  representative: 

10  December  1944  28th  Infantry  Division 

1500  Hours 

3d  Battalion,  110th  Infantry  Regiment  reports  it  observed  an  8-man 
enemy  patrol  in  the  vicinity  of  coordinates  851581  at  1115  hours. 

Enemy  withdrew  eastward  and  crossed  river  in  a rubber  boat  which  was 
found  in  the  vicinity  of  854584. 

Octant  4:  + Accuracy  - Relevance  - Directness.  Reports  4,  8,  12,  14, 

29,  32,  34,  and  36  are  located  in  this  octant.  The  commonality  of  these 
reports  is  that  the  analyst-subjects  regarded  them  as  accurate  but  of 
9uch  minor  nature  along  the  28th  and  4th  Division  fronts  as  not  to 
constitute  a relevant  threat. 

The  activity  consisted  of  small  patrols  which  were  not  aggressive, 
light  small  arms  and  artillery  fire,  occasional  aerial  reconnaissance, 
one  dead  German  with  a map,  and  a report  that  enemy  artillery  had  been 
kept  well  back  from  the  front  during  the  week  and  had  not  even  been 
firing  at  usual  targets. 

Subjects  apparently  analyzed  the  reports  In  much  the  same  way  as  the 
actual  ones  did  in  December  1944. 

They  missed  the  point  that  the  dead  German  soldier  (Report  29)  had 
a map  of  the  area  in  which  he  was  found  but  that  his  unit  was  last 
reported  much  farther  north,  which  should  have  led  them  to  question 
whether  his  unit  had  moved  south,  which,  in  turn,  could  have  been 
associated  with  other  southward  movements  farther  back  in  the  enemy 
rear  areas  reported  by  aerial  observers. 
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In  addition,  the  subjects  were  apparently  taken  in  by  the  artillery 
positions  and  relatively  small  amount  of  enemy  artillery  fire  in  the 
past  week  (Report  32). 

This  is  understandable  since  it  fitted  the  general  Allied  impression 
of  the  time  that  the  Germans  were  in  a defensive  posture  and  short  of 
ammunition.  However,  in  this  case  it  was  part  of  German  deception  built 
on  a past  reality  and  our  continuing  belief  in  it  abetted  by  German 
efforts  to  preserve  the  impression. 

The  text  of  Report  14  is  typical: 

13  December  1944  28th  Infantry  Division 

0400  Hours 

28th  Division  Artillery  reports  that  at  0300  hours  the  229th  Field 
Artillery  Battalion  detected  a two  (2)  man  enemy  patrol  in  the  vicinity 
of  its  CP.  The  patrol  withdrew  when  fired  on. 

Octant  5:  - Accuracy  + Relevance  + Directness.  Reports  5,  16,  21,  31, 

and  38  are  located  in  this  octant.  Analysts  correctly  evaluated  the 
relevant  and  dangerous  nature  of  these  reports,  if  true:  increased 
numbers  of  troops  in  towns  near  the  front  lines,  recall  of  men  from  rear 
area  schools  to  front  line  units  for  an  attack;  forward  moves  of  a 
division  CP;  units  shifting  along  the  front,  and  three  new  reserve 
divisions  in  the  enemy  rear  area. 

However,  they  placed  no  credence  in  POW  reports,  nor  even  in  those 
of  civilians  characterized  by  local  police  as  reliable.  Additionally, 
many  may  have  recognized  Report  21  as  really  false.  If  they  checked 
their  maps  and  knew  their  tactics  they  realized  that  the  place  reported 
as  the  new  location  of  the  26th  VG  Division  Command  Post  was  entirely 
too  close  to  the  FEBA. 

The  text  of  Report  16  is  representative: 

13  December  1944  28th  Infantry  Division 

2100  Hours 

4th  Infantry  Division  reports  that  a deserter  from  the  2d  Battalion, 
316th  Regiment,  212th  Volksgrenadier  Division  stated: 

a.  The  320th  Regiment  of  the  Division  was  on  line  for  about  two 
weeks  but  is  now  in  reserve. 
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b.  Approximately  600  men  on  special  types  of  duty  in  rear  areas 
were  called  back  to  the  316th  and  423rd  Regiments  of  the  212th 
Volksgrenadier  Division  by  a secret  order  on  11  December.  They 
were  told  to  return  to  their  regiments  immediately  because  they 
were  needed  for  an  upcoming  attack  against  the  thinly  held 
American  line.  These  men  for  the  most  part  had  been  attending 
non-commissioned  officer  schools. 

c.  Morale  of  German  troops  in  his  division  is  good  to  very  good. 

Octant  6:  - Accuracy  4-  Relevance  - Directness.  Reports  6,  13,  and  22 
are  within  this  octant.  The  rationale  for  this  grouping  is  hard  to 
explain.  Two  reports  (#6  and  #13)  are  of  major  importance.  The  third 
report  (#22)  is  rather  minor  and  seems  out  of  place. 

Move  of  an  Army  Headquarters  southward  (Report  #6)  is  significant 
even  though  the  withdrawal  across  the  Rhine  appears  unlikely  for  the 
reasons  stated.  The  relevant  threat  was  recognized  if  the  report  were 
true.  The  POW  was  probably  not  considered  reliable,  and  rightly  the 
withdrawal  across  the  Rhine  was  not  believed. 

Report  #13  also  represents  a relevant  threat  since  it  indicates  a 
buildup  of  at  least  three  divisions  opposite  VIII  Corps  and  First  Army. 
However,  at  least  part  of  the  report  originated  with  a POW,  and  from  the 
position  on  the  west  bank  of  the  Moselle,  the  three  divisions  could  be 
used  against  the  north  flank  of  the  Third  Army  to  the  south  as  well  as 
against  First  Army. 

Apparently,  for  that  reason,  the  subjects  did  not  believe  the 
report  accurate  or  infer  much  from  it. 

On  the  other  hand,  Report  #22  is  a straightforward  front  line 
report  of  hearing  enemy  vehicles  three  times  in  the  early  morning, 
although  conditions  were  such  that  the  direction  or  size  of  the  movement 
could  not  be  determined.  One  would  have  expected  such  a report  to  fall 
into  Octants  3 ( + - + ) or  4 (+--)  rather  than  into  Octant  6 
(-+-). 

The  text  of  Report  6 which  is  a false  message  added  to  the  data  set 

was : 

11  December  1944  28th  Infantry  Division 

1200  Hours 

106th  Infantry  Division  reports  a deserter  who  surrendered  9 December 
says  he  is  from  Headquarters  15th  Army  which  recently  moved  from  the 
Rotterdam  area  to  a location  in  the  Cologne-Bonn-Dueren  area.  Rumor  at 
his  Headquarters  is  that  Field  Marshal  von  Rundstedt  has  ordered  with- 
drawal to  the  East  bank  of  the  Rhine  because  shortage  of  reserves  will 
make  defense  west  of  the  Rhine  no  longer  feasible  after  the  first  of  the 
year. 


A 
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Octant  7:  - Accuracy  - Relevance  + Directness.  Reports  1 and  23  are 

located  in  this  octant.  Both  of  these  reports  originated  with  deserters, 
whom  the  subjects  apparently  do  not  trust,  and  the  information  reported 
consists  of  minor  locations  of  only  local  interest. 

Report  #23  is  false  because  the  deserter  says  he  was  bom  in 
Strasbourg  which  is  in  the  Franco-German  border  area  and  all  such  soldiers 
had  been  screened  out  of  the  German  front  line  units  during  November  and 
December  1944  to  prevent  just  such  desertions. 

The  test  of  Report  23  was  as  follows: 

14  December  1944  28th  Infantry  Division 

1015  Hours 

1st  Battalion,  110th  Infantry  Regiment  reports  that  an  enemy  deserter 
picked  up  at  132300  hours  in  TINTESMUEHLE , a small  village  at  coordinates 
850673  at  the  bend  in  the  river  east  of  HEINERSCHEID  (821675)  and  SE  of 
KALBORN  (840682)  says  he  was  bom  in  STRASBOURG.*  He  stated  that  the 
machine  gun  covering  the  ruins  of  the  bridge  at  coordinates  850672  was 
moved  on  7 December  to  a better  location  100  meters  from  its  former 
position  where  it  can  fire  directly  down  the  center  line  of  the  old 
bridge . 

Octant  8:  - Accuracy  - Relevance  - Directness.  Reports  3,  15,  and  35 

are  located  in  this  octant.  In  content,  all  of  these  messages  have  infor- 
mation which  is  of  minor  importance  or  irrelevant  to  the  28th  Division. 

In  two  out  of  three  cases,  it  is  one  day  to  15  days  late. 

Report  #15  is  false,  since  a patrol  from  the  3d  Battalion,  110th 
Infantry  Regiment  would  not  have  been  that  far  south  without  an  expla- 
nation having  been  given,  and  Report  #35  is  false  since  it  is  unlikely 
that  the  Germans  would  have  let  a railroad  employee  visit  relatives 
behind  Allied  lines  just  before  the  counteroffensive. 

It  appears  that  the  subjects  correctly  analyzed  these  reports  except 
for  #3,  with  respect  to  Accuracy.  They  had  no  apparent  basis  for  calling 
the  report  from  the  3d  Battalion,  110th  Infantry  Regiment  inaccurate. 

The  text  of  Report  35,  still  another  fake  message,  was  as  follows: 

15  December  1944  28th  Infantry  Division 

0600  Hours 

4th  Infantry  Division  Interrogation  of  Prisoner  of  War  Team  report 
states  that  a German  national  from  SAARBRUECKEN**  was  picked  up  while 


*On  West  Bank  of  the  Rhine,  Southeast  of  the  VII  Corps  sector. 
♦♦Opposite  Third  Army  Zone  to  the  Southeast. 
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visiting  relatives  vicinity  Luxembourg  City  on  6 December.  He  claims  he 
is  an  employee  of  the  German  Railroad  System.  He  further  states  that  on 
30  November  the  marshalling  yards  in  ZWEIBRUECKEN*  were  tied  up  for  six 
hours  when  two  carloads  of  explosives  collided  and  exploded. 

To  this  point,  it  has  been  established  that  the  judgmental  space 
employed  by  the  subjects  in  evaluating  the  sample  of  intelligence  reports 
is  coherent  and  produces  groupings  of  the  reports  which  reflect  the 
nature  of  the  content  of  the  judgmental  spaces. 

Although  the  examination  of  the  reports  in  each  of  the  octants  of 
the  combined  judgment  space  may  have  raised  some  questions  about  the 
value  of  the  RELEVANCE  and  DIRECTNESS  dimensions,  it  is  clear  that  the 
ACCURACY  dimension  plays  a major  role  in  the  subjects'  evaluation  of  the 
reports . 


*0pposite  Third  Army  Zone  to  the  Southeast. 
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Figure  B-1.  Report  Attribute  Structure:  Untrained  Subjects 


Figure  b / neport  «rrriouie  structure:  Trained  Subjects 


Figure  B 3.  Report  Attribute  Structure:  Combined  Subject  Groups 
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APPENDIX  C 

ANALYSIS  OF  RELATIONSHIPS  AMONG  ACCURACY,  RELIABILITY, 

AND  OTHER  SCALES 

The  polar  coordinate  plots  of  Figures  C-l  through  C-9  in  which  the 
correlations  of  the  Reliability  (Scale  47),  Accuracy  (Scale  48)*,  and 
combined  Global  ratings  (Scale  51)  with  all  other  report  characteristics 
are  depicted,  are  designed  to  display  the  patterns  of  zero-order 
relationships  for  these  particular  scales. 

Each  5-degree  radius  of  the  plots,  starting  at  zero  degrees  (center, 
top)  through  255  degrees,  represents,  in  order,  the  50  scales  plus  a 
derived  51st  scale  which  is  the  average  of  the  two  separate  global 
ratings  (Nos.  49  and  50).  The  center  of  the  plot  represents  either 
the  Reliability  'Scale  47) , the  Accuracy  (Scale  48) , or  Combined 
(average)  Global  (Scale  51)  ratings. 

The  length  of  the  polar  vector  outward  from  the  scale  used  as  the 
basis  of  comparison  expresses  the  degree  of  relationship  to  that  base 
scale  as  indexed  by  the  Pearson  Product  Moment  correlation  coefficient. 
The  farther  the  end  of  the  scale  vector  from  the  center  of  the  figure, 
the  greater  the  relationship  between  the  scales. 

The  plots  display  only  correlations  accounting  for  more  than  half  of 
the  correlating  variance.  No  vectors  have  been  drawn  for  scales  whose 
correlation  with  the  base  scale  is  less  than  an  absolute  value  of  .70. 
This  value  was  arbitrarily  chosen  so  as  to  select  only  those  scales  with 
the  greatest  predictive  relationships. 

The  correlations  displayed  in  these  plots  have  been  derived  by  using 
subject  averages  as  entries  and  thus  reflect  the  degree  of  relationship 
between  each  scale  with  respect  to  the  similarity  of  dissimilarity  in  the 
ratings  of  the  40  reports. 

Thus,  a high  correlation  between  two  scales  indicated  that  the 
average  subject  of  his  particular  group  tended  to  rate  the  reports  as 
being  similar  with  respect  to  those  qualities. 

For  example,  if  Reports  1,  6,  10,  and  30  were  rated  by  the  average 
subject  as  being  very  accurate  and  these  same  reports  were  also  rated 
as  being  highly  reliable,  then  the  correlation  between  Accuracy  and 
Reliability  would  be  high. 


*See  Figure  4 for  the  scales  corresponding  to  the  numbered  identifications. 
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Reliability  Relationships.  Figures  C-l  through  C-3  display  the 
relationships  between  the  Reliability  (Scale  47)  and  all  other  scales. 

Inspection  of  the  figures  reveals  that  scales  rating  True-False 
(Scale  2),  Dependable-Undependable  (Scale  17),  Truthful-Deceptive  (Scale 
21),  Accurate-Erroneous  (Scale  31),  and  Accuracy  (Scale  48)*  all  have 
correlations  in  excess  of  .90  for  the  untrained  subject  group. 

The  trained  subjects  produced  highest  correlations  with  reliability 
for  the  scales  rating  True-False  (Scale  2),  Dependable-Undependable 
(Scale  17),  Accuracy  (Scale  48),  Global-1  (Scale  49),  Global-2  (Scale 

50) .  For  the  combined  subjects.  Reliability  was  most  closely  associated 
with  the  ratings  of  Dependable-Undependable  and  Accuracy. 

Accuracy  Relationships.  Figures  C-4  through  C-6  display  the  relation- 
ships between  Accuracy  (Scale  48)  ratings  and  all  other  scalar  qualities. 

Inspection  of  the  figures  indicates  that  Accuracy  correlates  beyond 
.90  with  True  (Scale  2)  (opposites  henceforth  dropped).  Dependable 
(Scale  17),  Truthful  (Scale  21),  Faultless  (Scale  29),  Acceptable 
(Scale  38),  Reliability  (Scale  47),  and  all  Global  scales  (49,  50  and 

51) . 


The  trained  subjects  also  disolaved  highest  relationships  between 
Accuracy  and  the  above  scales  excepting  Dependable,  Faultless  (Scale  29), 
and  Acceptable  (Scale  38) . 

The  combined  groups  produced  highest  relationships  between  Accuracy 
and  True,  Dependable,  Truthful,  Faultless  (Scale  29),  Accurate  (Scale 
31),  Confirmed  (Scale  40),  Reliability,  and  all  Global  ratings  (Scales 
49 , 50 , and  51) . 

Global  Relationships.  Figures  C-7  through  C-9  display  the  Combined 
Glooal  (Scale  51)  relationships.  The  untrained  subject  group  rated 
True  (Scale  2) , Accuracy  (Scale  48) , Global-1  (Scale  49)  and  Global-2 
(Scale  50)  as  most  closely  associated  with  the  Combined  Global  variable. 

Obviously  the  high  correlations  between  the  separate  and  combined 
global  ratings  are  considered  as  instrument  relationships  and  should  be 
discounted . 

The  trained  subject  group  correlations  indicate  highest  relationships 
between  the  Combined  Global  score  and  True,  Truthful,  Accurate,  Reli- 
ability, Accuracy  and  the  separate  Global  ratings. 


*Note  that  the  intelligence  ratings  as  traditionally  employed  are 
referred  to  as  Accuracy  and  Reliability.  The  quality  scales,  on  the 
other  hand,  are  referred  to  as  Accurate-Erroneous  and  Reliable-Variable. 
See  Figure  4. 
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The  combined  subject  groups  display  highest  relationships  with  True, 
Dependable,  Truthful,  Faultless,  Accurate,  Confirmed,  Accuracy,  and  the 
separate  Globals. 

Cluster  Relationships.  Figure  C-10  displays  the  cluster  groupings 
around  the  Accuracy,  Reliability  and  Global  variables  of  the  scalar 
qualities  for  each  of  the  subject  groupings. 

For  the  purposes  of  this  display,  that  scale  having  the  highest 
correlation  with  each  of  the  primary  variables  is  identified  by  the  arrow 
from  it  to  the  next  variable. 

The  correlation  matrix  is  then  searched  for  that  variable  which  has 
the  highest  relationship  with  this  second  variable  and  so  on  until  one 
of  the  subsequent  variables  has  its  highest  correlation  with  one  of  the 
previously  identified  variables  already  displayed. 

Such  clusters  can  be  interpreted  as  the  most  conservative  ordering 
of  the  scale  relationships  since,  at  each  successive  step,  only  the 
highest  correlation  with  each  variable  is  considered. 

All  correlations  of  these  clusters  excepting  that  for  the  untrained 
subjects  with  respect  to  the  correlation  between  Scales  51  (Combined 
Global)  and  49  (Global-1)  (its  value  being  .91),  are  in  fact,  in  excess 
of  .94. 

These  analyses  of  the  zero-order  relationships  indicate  quite  clearly 
that  the  Reliability  and  Accuracy  ratings  are  highly  correlated  in  both 
the  trained  and  untrained  analyst  groups.  For  the  total  subject  group, 
the  correlation  is  in  excess  of  .95. 

Reliability  is  being  assessed  by  these  subjects  as  if  it  were  a 
measure  of  the  truthfulness,  acceptability  and  accuracy  of  the  intelli- 
gence message.  The  training  of  these  subjects  does  not  change  the 
pattern  significantly. 

Also,  with  respect  to  the  traditionally  employed  scales  Reliability 
and  Accuracy,  Accuracy  is  the  better  single  measure.  The  judged 
Reliability  of  the  message  is  being  determined  by  the  judged  Accuracy 
rather  than  vice  versa. 

To  some  estent,  this  result  was  inherent  in  the  test  given  the 
subjects.  Accuracy  is  a one-time  evaluation  of  each  report,  but 
Reliability  of  source  or  agency,  if  determined  correctly,  can  be  built 
up  only  over  a period  of  time  as  the  analyst  works  with  the  reports  of 
each  source  or  agency  and  develops  an  appreciation  of  their  reliability 
from  opportunities  which  arise  to  check  on  the  information  as  reported. 

This  fact  in  itself  indicates  how  dependent  Reliability  is  on  the 
average  Accuracy  of  reports  from  any  given  source  or  agency  submitted 
over  a period  of  time. 
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In  the  test  situation,  the  subjects  were  initially  forced  to  infer 
Reliability  from  other  qualities  of  each  report  or  to  make  assumptions 
about  it  based  on  the  nature  of  the  source  or  agency,  e.g.,  patrols, 
subordinate  units  G2s  and  S2s,  the  Divisional  IPW  team,  G2s  of  adjacent 
divisions  or  higher  headquarters.  Air  Corps  pilots,  friendly  and  enemy 
civilians  (except  in  those  cases  in  which  local  police  made  an  evalua- 
tion), friendly  local  police,  POWs  and  deserters. 

However,  despite  this  difficulty,  findings  of  this  study  correspond 
to  those  of  earlier  studies  of  the  traditional  Accuracy  and  Reliability 
Scales  by  indicating  a high  degree  of  inter-correlation  in  their  use 
(see  Samet,  1975). 

Accuracy  ratings  are  apparently  well  assessed  by  the  Global  scale 
ratings.  Significantly,  the  Global  Scale  rating  appears  to  be  a purer 
measure  of  the  accuracy  of  the  messages  than  is  the  standard  Accuracy 
measurement . 

Examination  of  Figure  C-10  indicates  that  the  cluster  chains  for  the 
Global  ratings  do  not  extend  beyond  their  own  sub-components . 

Although  Reliability  does  correlate  with  the  average  Global  ratings 
in  the  combined,  trained  and  untrained  subject  samples,  the  A-F  Reli- 
ability ratings  are  not  as  highly  correlated  with  the  Global  ratings  as 
they  are  with  the  1-6  Accuracy  judgments. 

The  conclusion  is  straightforward.  The  Reliability  ratings,  as  taken 
from  this  sample  of  subjects  with  this  set  of  messages,  were  nearly  use- 
less and  the  accuracy  ratings  are  more  purely  and  thus  better  measured 
by  the  Global  rating. 

Considering  that  fewer  instructions  are  required  for  use  of  the 
Global  scale  compared  to  those  required  for  use  of  the  Accuracy  scale, 
the  use  of  Global  ratings  instead  of  the  present  Reliability  and  Accuracy 
ratings  might  improve  the  quality  of  the  initial  ratings  of  information 
accuracy  as  well  as  subsequent  understanding  of  the  meaning  of  the 
ratings  (Samet,  1975). 
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APPENDIX  D 

STEP  MULTIPLE  REGRESSION  ANALYSES 

The  following  discussion  presents  still  another  way  of  assessing  the 
judgmental  characteristics  of  the  Accuracy  and  Reliability  ratings.  In 
what  follows,  we  shall  be  concerned  with  the  degree  and  quality  of  the 
Accuracy  and  Reliability  ratings  as  predicted  by  their  multiple  relation- 
ships with  the  other  scales  employed. 

In  these  analyses,  we  shall  consider  the  patterns  of  ratings  which 
mutually  predict  the  two  forms  of  ratings  as  criteria  with  respect  to 
both  subject  and  report  averages. 

Analyses  Employing  Subjects  as  Observations.  As  indicated  in  the 
statistical  procedures  of  Figure  3 (Analyses  IV:  A through  C) , these 
analyses  were  made  by  averaging  over  the  individual  reports  for  each 
subject's  scale  judgments. 

These  mean  report  observations  on  each  subject  and  scale  were 
multiply  intercorrelated  over  the  subject  scores  against  the  success- 
ively isolated  criterion  variables  of  Reliability  (Scale  47) , Accuracy 
(Scale  48),  and  Combined  Global  (Scale  51)*  ratings  for  each  of  the 
separately  considered  subject  groupings. 

These  combinations  produced  a total  of  nine  separate  scale  analyses; 
three  subject  groupings  by  three  criterion  variables.  The  summary 
results  of  each  of  these  multiple  regression  analyses  are  displayed  in 
Tables  D1  through  D3. 

All  displays  record  the  order  in  which  each  new  variable  was  added  to 
the  criterion  predictive  set  and  the  resultant  multiple  correlation  of 
the  total  predictive  set  after  each  step.  Variables  are  added  to  the 
equation  at  each  step  so  as  to  maximize  the  multiple  R. 

The  first  step  of  each  display  records  that  zero-order  relationship 
between  the  criterion  and  the  single  predictor  variable  with  greatest 
correlation.  Subsequent  steps  are  multiple  correlations  between  the 
criterion  and  the  cumulative  set  of  predictors. 

When  subjects  are  used  as  observations  for  the  correlations  among 
scale  qualities,  the  interpretation  of  these  multiple  Rs  is  different 
from  that  whqn  reports  are  used  as  observations,  as  discussed  below. 

*In  all  of  the  following  regression  analyses,  the  two  separate  Global 
ratings  have  been  combined  into  a single  variable  by  averaging  their 
separate  values.  The  new  variable  is  numbered  Scale  51  (see  also 
discussion  above),  and  Scales  49  and  50  corresponding  to  Global-1  and 

Global-2  have  been  dropped. 
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Here,  the  correlations  reflect  the  degree  to  which  each  subject, 
without  regard  to  report  ratings,  tends  to  use  the  scale  attributes  in 
the  same  manner.  We  have  to  ask  "Is  there  a consistent  pattern  of  scale 
usage  across  differing  subjects?"  and  "Do  differing  subjects  use  the 
scale  attributes  similarly  or  differently?" 

The  answer  to  these  questions  is  expressed  by  the  maximum  value  of 
the  multiple  R achieved  at  the  final  step  of  the  analysis.  If  subjects 
tend  to  have  report-averaged  ratings  on  one  scale  which  multiply  covary 
with  other  scales,  the  multiple  R will  be  high. 

An  inspection  of  the  results  indicates  that  the  maximum  multiple  R 
achieved  after  entering  15  variables  is  uniformly  equal  to  or  greater 
than  .90  for  all  analyses. 

That  means  that  something  more  than  80%  of  the  total  subject  rating 
variance  in  the  criterion  variables  can  be  accounted  for  by  on  15  of  the 
scalar  qualities.  It  remains  to  determine  only  the  particular  scalar 
qualities  in  each  analysis  which  are  determining  that  variance. 

With  Reliability  as  criterion  variable  and  considering  only  the 
combined  subject  sample,  more  than  half  the  obtained  subject  variability 
in  the  reliability  ratings  is  accounted  for  by  the  Accuracy  and  Global 
scales.  The  remaining  variance  of  the  Reliability  ratings  is  accounted 
for  by  the  scales : 

Appropriate  (20),  Accurate  (31),  Possible  (26),  Active  (1),  Hazardous 
(22),  Timely  (3),  Dependable  (17),  Expected  (23),  Extraordinary  (6), 
Widespread  (30),  Heavy  (25),  Truthful  (21),  and  Massive  (15)*  in  that 
order,  in  short,  by  the  determinants  of  Threat  Potential,  Accuracy, 
Expectedness  and  Dependability. 

With  Accuracy  as  the  criterion  variable,  more  than  80%  of  the  total 
variance  of  the  combined  subject  sample  is  accounted  for  by  the  qualities 
of  Reliability,  Acceptableness,  Accuracy,  Activeness,  and  Probableness 
(Table  D2). 

The  scales,  in  order,  are  Reliability  (47),  Acceptable  (38), 

Confirmed  (40),  Accurate  (31),  Faultless  (29),  Massive  (15)  Useful  (18), 
Active  (1) , Probable  (43) , Possible  (26) , True  (2) , Dependable  (17) , 
Extraordinary  (6) , Direct  (11)  and  Relevant  (33) . 


*0nly  one  pole  of  the  bipolar  pairs  is  used  and  it  has  been  reflected 
to  common  direction  (see  Figure  4) . 
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With  the  average  Global  rating  (Scale  51)  as  the  criterion  variable, 
more  than  80%  of  the  total  variance  of  the  combined  subject  sample  is 
accounted  for  by  the  characteristics  of  Accuracy,  Intelligibility, 
Massiveness,  Consistency,  and  Usefulness  (Table  D-3) . 

The  scales,  in  order,  are:  True  (2),  Intelligible  (7),  Reliability 
(47),  Probable  (43),  Direct  (11),  Clear  (28),  Interpreted  (32),  Wide- 
spread (30),  Extraordinary  (6),  Consistent  (45),  Useful  (18),  Massive 
(15),  Precarious  (5),  Faultless  (29),  and  Truthful  (21). 

Analyses  Employing  Reports  as  Observations.  For  the  purposes  of 
these  analyses  each  report  was  treated  as  a separate  observation  by 
taking  means  across  the  appropriate  subject  groupings  on  each  of  the 
rating  scales  (see  Tables  D-4  through  D-6) . 

Such  analyses  assess  the  multiple  correlations  of  the  scale  qualities 
with  respect  to  the  separately  considered  reports.  Thus,  patterns  of 
inter-relationships  indicate  the  degree  to  which  the  characteristics  of 
the  reports  tend  to  covary. 

These  analyses  are  in  a sense  more  conservative  explanations  of  the 
scalar  structure  than  the  previous  subject  analyses.  Only  the  report 
variance  in  scale  usage  need  be  accounted  for  instead  of  the  larger 
subject  variance. 

Reports  which  are  judged  by  the  averaged  subject  sample  to  have  high 
ratings  on  more  than  one  attribute  will  show  high  correlation  with  those 
other  attributes. 

The  multiple  Rs  of  these  analyses  are  uniformly  higher  than  in  the 
previous  analyses.  The  15  variables  for  each  of  the  criterion  variables 
across  the  separate  and  combined  subject  groupings  account  for  either 
all  (R  ■ 1.00)  or  very  nearly  all  (R  = .99)  of  the  report  variance 
obtained  with  respect  to  these  attributes. 

In  point  of  fact,  the  zero-order  correlations  of  the  first  step  of 
each  of  these  analyses  contribute  more  than  81%  of  the  criterion 
variance  alone. 

For  the  trained,  untrained,  and  combined  subject  groupings,  the 
single  best  predictor  of  the  reliability  ratings  is  Accuracy  (Scale  48) 
(Table  D-4) . Dependable  (Scale  17)  and  Consistent  (Scale  45)  improve  the 
determination  for  the  combined  subjects,  with  Dependable  being  the 
second  highest  determinant  in  the  separate  groupings. 

All  subject  groupings  indicate  that  Reliability  (Scale  47)  and 
Global  (Scales  49  and  50  or  51)  ratings  are  the  two  highest  determinants 
of  the  Accuracy  ratings  when  considered  with  respect  to  report  variance. 
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The  single  best  predictor  of  the  Combined  Global  rating  is  Accuracy 
(Scale  48)  for  all  subject  groupings,  with  percent  of  accounted  variance 
in  excess  of  81%. 

In  summary  of  these  multiple  Regression  Analyses,  it  can  be  said  that 
they  are  consistent  with,  confirm,  and  extend  the  earlier  conclusions 
regarding  the  traditionally  employed  scales  of  Accuracy  and  Reliability. 

It  can  be  seen  that  the  Global  rating  is  a purer  assessment  of  what 
is  being  measured  by  the  Accuracy  rating;  Accuracy  judgments  being 
heavily  influenced  by  the  subjects'  Reliability  assessments.  Reliability, 
on  the  other  hand,  is  influenced  by  other  attributes. 

The  Reliability  assessments  of  the  reports  employed  is  significantly 
based  on  an  assessment  of  the  internal  consistency  of  the  messages  and 
the  likelihood  of  their  content. 

With  respect  to  report  variance,  Accuracy  and  Reliability  ratings  are 
each  predicted  by  the  other,  for  both  trained  and  untrained  subjects. 

For  the  Reliability  ratings,  the  single  best  predictor  (R  =*  .94)  is 
Accuracy  in  both  groups  of  subjects. 

Accuracy  is  also  predicted  by  Reliability  in  both  subject  groups  with 
R = .94.  On  the  other  hand,  the  multiple  predictions  of  Reliability  and 
Accuracy  analyzed  with  respect  to  subject  differences  indicate  that 
Reliability  judgments  are  predicted  only  by  the  Accuracy  ratings  given 
by  the  untrained  subjects  and  not  by  the  trained  subjects. 

For  the  trained  subjects,  the  individual  ratings  of  Accuracy  can  be 
predicted  from  their  ratings  of  Reliability,  while  Reliability  ratings 
are  not  predicted  by  the  Accuracy  ratings  of  the  trained  subjects. 

Because  of  the  nature  of  the  simulation  of  battlefield  conditions,  or 
for  that  matter  any  simulation,  the  subjects  could  not  have  had  any 
direct  data  on  which  to  base  their  Reliability  assessments.  Nonetheless, 
two  paradoxical  findings  emerge. 

First,  the  school-trained  subjects'  judgments  of  report  Accuracy  can 
be  predicted  with  high  precision  from  their  judgments  of  the  Reliability 
of  the  sources  and  agencies,  second,  the  assessments  of  Reliability,  as 
made  by  all  subjects,  indicated  that  on  the  average  the  reports  were 
Judged  to  be  highly  Reliable  (see  Table  4). 

It  appears  clear  that  the  two  scales  are  at  least  partly  contaminated 
by  one  or  both  of  their  other  measurements  and  are  susceptible  to 
misinterpretation  and  misuse. 
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Table  D-l.  Step  Multiple  Correlations  Using  Subjects 
as  Observations  with  Reliability  as  Criterion 

RELIABILITY  (1*7):  SUBJECT  MEANS 


Subject  Sample 

r 

Combined 

Trained 

Untrained 

Step 

Scale 

R 

Scale 

R 

Scale 

R 

1 

48 

(Accuracy) 

.64 

1 

(Active) 

.47 

48 

(Accuracy) 

.82 

2 

51 

(Combined 

Global) 

.71 

43 

(Probable) 

.61 

31 

(Accurate) 

.84 

3 

20 

(Appropriate) 

.74 

31 

(Accurate) 

.73 

3 

(Timely) 

.88 

4 

31 

(Accurate) 

.78 

15 

(Massive) 

.81 

23 

(Expected) 

.90 

5 

26 

(Possible) 

.80 

35 

(Analyzable) 

.85 

21 

(Truthful) 

.91 

6 

1 

(Active) 

.82 

36 

(Routine) 

.89 

18 

(Useful) 

.92 

7 

22 

(Hazardous) 

.84 

24 

(Impl ied) 

.93 

37 

(Completed) 

.93 

8 

3 

(Timely) 

.85 

41 

(Constant) 

.95 

16 

(Substant iated) 

.94 

9 

17 

(Dependable) 

.86 

12 

(Variable) 

.96 

26 

(Possible) 

.94 

10 

23 

(Expected) 

.88 

5 

(Precarious) 

.97 

46 

(Dangerous) 

.95 

11 

6 

(Extra- 

ordinary) 

.89 

11 

(Direct) 

.98 

11 

(Direct) 

.96 

12 

30 

(Widespread) 

.90 

28 

(Clear) 

.99 

51 

(Combined 

Global) 

.96 

13 

25 

(Heavy) 

.91 

21 

(Truthful) 

1.00 

7 

(Intelligible) 

.97 

14 

21 

(Truthful) 

.93 

8 

(Feasible) 

1.00 

17 

(Dependable) 

.97 

15 

15 

(Massive) 

.93 

29 

(Faultless) 

1.00 

29 

(Faultless) 

.98 
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Table  D-2.  Step  Multiple  Correlations  Using  Subjects 
as  Observations  with  Accuracy  as  Criterion 

ACCURACY  (48) : SUBJECT  MEANS 


Subject  Sample 

Combined 

Trained 

Untrained 

Step 

Scale 

R 

Scale 

R 

Scale 

R 

1 

47 

(Reliability) 

.64 

29 

(Faultless) 

.58 

47 

(Reliability) 

.82 

2 

38 

(Acceptable) 

.74 

47 

(Reliability) 

.66 

3 

(Timely) 

.88 

3 

40 

(Confirmed) 

.76 

1 

(Active) 

.73 

40 

(Confirmed) 

.89 

m 

31 

(Accurate) 

.79 

31 

(Accurate) 

.79 

15 

(Massive) 

.91 

M 

29 

(Faultless) 

.81 

18 

(Useful) 

.89 

7 

(Intelligible) 

.92 

fl 

15 

(Massive) 

.82 

22 

(Hazardous) 

.93 

21 

(Truthful) 

.93 

B 

18 

(Useful) 

.83 

38 

(Acceptable) 

.94 

35 

(Analyzable) 

.94 

H 

1 

(Active) 

.84 

43 

(Probable) 

.98 

46 

(Safe) 

.95 

n 

43 

(Probable) 

.85 

23 

(Expected) 

.98 

22 

(Hazardous) 

.95 

10 

26 

(Possible) 

.86 

20 

(Appropriate) 

.99 

13 

(Unde  rs  tandab le) 

.96 

n 

2 

(True) 

.87 

15 

(Massive) 

.99 

14 

(Likely) 

.96 

12 

17 

(Dependable) 

.88 

13 

(Understand- 

able) 

1.00 

33 

(Relevant) 

.96 

13 

6 

(Extra- 

ordinary) 

.89 

41 

(Constant) 

1.00 

37 

(Completed) 

.97 

14 

11 

(Direct) 

.89 

51 

(Combined 

Global) 

1.00 

51 

(Combined 

Global) 

.97 

15 

33 

(Relevant) 

.90 

44 

(Pertinent) 

1.00 

5 

(Precarious) 

.98 
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Table  D-J.  Step  Multiple  Correlations  Using  Subjects  as 
Observations  with  Combined  Global  Ratings  as  Criterion 

COMBINED  GLOBAL  % (49-50):  SUBJECT  MEANS 


Step 

Combined 

Scale 

R 

Subject  Sample 

Trained 

Scale  R 

Untrained 

Scale 

R 

1 

2 

(True 

.72 

2 

(True) 

.74 

21 

(Truthful 

.71 

2 

7 

(Intelligible) 

.76 

32 

(Interpreted' 

.81 

43 

(Probable) 

.77 

3 

47 

(Reliability) 

.78 

8 

(Feasible) 

.84 

6 

(Ordinary) 

.80 

4 

43 

(Probable) 

.80 

9 

(Factual) 

.86 

47 

(Reliability) 

.83 

5 

11 

(Direct) 

.82 

43 

(Probable) 

.89 

37 

(Completed) 

.86 

6 

28 

(Clear) 

.83 

37 

(Completed) 

.92 

1 

(Active) 

.87 

7 

32 

(Interpreted) 

.84 

29 

(Faultless) 

.94 

39 

(Many) 

.88 

8 

30 

(Widespread) 

.85 

16 

(Substan- 

tiated) 

.97 

11 

(Direct) 

.89 

9 

6 

(Extra- 

ordinary) 

.87 

38 

(Acceptable) 

.98 

15 

(Massive) 

.90 

10 

45 

(Consistent) 

.87 

13 

(Understand- 

able) 

.99 

24 

(Implied) 

.92 

11 

18 

(Useful) 

.88 

17 

(Dependable) 

.99 

3 

(Timely) 

.93 

12 

15 

(Massive) 

.89 

10 

(Stable) 

.99 

28 

(Clear) 

.94 

13 

5 

(Precarious) 

.90 

12 

(Reliable) 

1.00 

34 

(Secure) 

.94 

14 

29 

(Faultless) 

.90 

30 

(Widespread) 

1.00 

42 

(Large  Scale) 

.95 

15 

21 

(Truthful) 

.91 

31 

(Accurate) 

1.00 

13 

(Understandable) 

.96 
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Table  D-4.  Step  Multiple  Correlations  Using  Reports  as 
Observations  with  Reliability  as  Criterion 

RELIABILITY  (47):  REPORT  MEANS 


Step 

Combined 

Scale 

R 

Subject  Sample 

Trained 

Scale  R 

Untrained 

Scale 

R 

1 

48 

(Accuracy) 

.95 

48 

(Accuracy) 

.96 

48 

(Accuracy) 

.94 

2 

45 

(Consistent) 

.96 

17 

(Dependable) 

.97 

17 

(Dependable) 

.95 

3 

17 

(Dependable) 

.98 

7 

(Intelligibl 

e) . 98 

23 

(Expected) 

.96 

4 

14 

(Likely) 

.98 

22 

(Hazardous) 

.99 

51 

(Combined 

Clobal) 

.97 

5 

16 

(Substan- 

tiated) 

.99 

14 

(Likely) 

.99 

44 

(Pert inent) 

.97 

6 

26 

(Poss ible) 

.99 

27 

(Observed) 

.99 

10 

(Stable) 

.97 

7 

23 

(Expected) 

.99 

9 

(Factual) 

.99 

14 

(Likely) 

.98 

8 

38 

(Acceptable) 

.99 

19 

(Volatile) 

.99 

31 

(Accurate) 

.98 

9 

51 

(Combined 

Clobal) 

.99 

43 

(Probable) 

.99 

32 

(Interpreted) 

.98 

10 

30 

(Widespread) 

.99 

11 

(Di rect) 

.99 

34 

(Secure) 

.98 

11 

28 

(Clear) 

.99 

38 

(Acceptable) 

1.00 

3 

(Timely) 

.99 

12 

7 

(Intelligible) 

.99 

31 

(Accurate) 

1.00 

6 

(Ordinary) 

.99 

13 

22 

(Hazardous) 

.99 

40 

(Confirmed) 

1.00 

41 

(Constant) 

.99 

14 

19 

(Volatile) 

.99 

8 

(Feasible) 

1.00 

9 

(Factual) 

.99 

15 

13 

(Understand- 

able) 

.99 

20 

(Appropriate 

)1.00 

37 

(Completed) 

.99 
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Table  D-“5.  Step  Multiple  Correlations  Using  Reports  as 
Observations  with  Accuracy  as  Criterion 

ACCURACY  (48) : REPORT  MEANS 


Subject  Sample 

Combined 

Trained 

Untrained 

Step 

Scale 

R 

Scale 

R 

Scale 

1 

51 

(Combined 

Global) 

.96 

51 

(Combined 

Global) 

.98 

47 

(Reliability) 

2 

47 

(Reliability) 

.99 

47 

(Reliability) 

.98 

51 

(Combined 

Global) 

3 

34 

(Secure) 

.99 

36 

(Routine) 

.98 

34 

(Secure) 

4 

22 

(Hazardous) 

.99 

30 

(Widespread) 

.99 

23 

(Expec  ted) 

5 

3 

(Timely) 

.99 

11 

(Direc  t) 

.99 

12 

(Reliable) 

6 

40 

(Confirmed) 

.99 

14 

(Likely) 

.99 

32 

(Interpreted) 

7 

23 

(Expected) 

.99 

ro 

(Appropriate) 

.99 

27 

(Observed) 

8 

12 

(Reliable) 

.99 

34 

(Secure) 

.99 

31 

(Accurate) 

9 

43 

(Probable) 

1.00 

8 

(Feasible) 

.99 

2 

(True) 

10 

5 

(Precarious) 

1.00 

19 

(Volatile) 

.99 

1 

(Active) 

11 

7 

( Intelligible)!. 00 

46 

(Safe) 

1.00 

30 

(Wi despread ) 

12 

28 

(Clear) 

1.00 

27 

(Observed) 

1.00 

3 

(Ti mely) 

13 

35 

(Analyzable) 

1.00 

16 

(Substan- 

tiated) 

1.00 

43 

(Probable) 

14 

17 

(Dependable) 

1.00 

45 

(Consistent) 

1.00 

42 

(Large  Scale) 

15 

9 

(Factual) 

1.00 

10 

(Stable) 

1.00 

35 

(Analyzable) 
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Table  D-6.  Step  Multiple  Correlations  Using  Reports  as 
Observations  with  Combined  Global  Ratings 
as  Criterion 

COMBINED  GLOBAL  51  (49-50):  REPORT  MEANS 


Step 

1 

Combined 

Scale  R 

Subject  Sample 

Trained 

Scale  R 

Untrained 

Scale  R 

48  (Accuracy)  .96 

48  (Accuracy)  .98 

48  (Accuracy)  .93 

2 

5 (Precarious)  .97 

2 (True)  .99 

6 (Ordinary)  .95 

3 

47  (Reliability)  .98 

13  (Understand-  .99 
able) 

14  (Likely)  .95 

4 

2 (True)  .98 

40  (Confirmed)  .99 

20  (Appropriate)  .96 

5 

16  (Substantiated) . 98 

19  (Volatile)  .99 

8 (Feasible)  .97 

6 

39  (Many)  .99 

46  (Safe)  .99 

12  (Reliable)  .97 

7 

8 (Feasible)  .99 

20  (Appropriate)  .99 

27  (Observed)  .98 

8 

3 (Timely)  .99 

18  (Useful)  .99 

32  (Interpreted)  .98 

9 

6 (Extra-  .99 

ordinary) 

4 (Specific)  .99 

43  (Probable)  .98 

10 

34  (Secure)  .99 

15  (Massive)  .99 

2 (True)  .98 

11 

5* , (Precarious)  .99 

30  (Widespread)  .99 

44  (Pertinent)  .98 

12 

36  (Routine)  .99 

11  (Direct)  .99 

28  (Clear)  .98 

13 

41  (Constant)  .99 

27  (Observed)  .99 

1 (Active)  .99 

14 

32  (Interpreted)  1.00 

12  (Reliable)  1.00 

36  (Routine)  .99 

15 

4 (Specific)  1.00 

13*  (Under-  1.00 

standable) 

26  (Possible)  .99 

♦Variable  dropped  from  predictor  set.  Such  variables  have  all  of  their 
reliable  variance  accounted  for  by  the  preceding  predictors. 
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1 HQDA  (DAFD  MFA) 

1 HQDA  (DARD  ARS-P) 

1 HQDA  (DAPC  PAS  A) 

1 HQDA  (DUSA-OR) 

1 HQDA  (DAMO  RQR) 

1 HQDA  (DASG) 

1 HQDA  (DA10-PI) 

1 Chief.  Consult  Div  (DA  OTSG).  Adelphi,  MD 
1 Mil  Asst.  Hum  Res.  ODDR&E,  OAD  (E&LS) 

1 HQ  USARAL,  APO  Seattle.  ATTN:  ARAGP-R 

1 HQ  First  Army.  ATTN:  AFKA-OI-TI 

2 HQ  Fifth  Army,  Ft  Sam  Houston 

1 Dir.  Army  Stf  Studies  Ofc,  ATTN:  OAVCSA  (DSP) 

1 Ofc  Chief  of  Stf.  Studies  Ofc 
1 DCSPER,  ATTN:  CPS/OCP 
1 The  Army  Lib,  Pentagon,  ATTN:  RSB  Chief 
1 The  Army  Lib,  Pentagon.  ATTN:  ANRAL 
1 Ofc.  Asst  Sect  of  the  Army  (R&D) 

1 Tech  Support  Ofc,  OJCS 
1 USASA,  Arlington,  ATTN:  IARD  T 

1 USA  Rsch  Ofc.  Durham,  ATTN:  Life  Sciences  Dir 

2 USARIEM.  Natick.  ATTN:  SGRD  UE  CA 

1 USATTC.  Ft  Clayton.  ATTN:  STETC  MO-A 
1 USAIMA.  Ft  Bragg.  ATTN:  ATSU  CTD-OM 
1 USAIMA.  rt  Bragg.  ATTN:  Marquat  Lib 
1 US  WAC  Ctr  & Sch.  Ft  McClellan,  ATTN:  Lib 
1 US  WAC  Ctr  & Sch.  Ft  McClellan.  ATTN:  Tng  Dir 
1 USA  Quartermaster  Sch.  Ft  Lee.  ATTN:  ATSM-TE 
1 Intelligence  Material  Dev  Ofc,  EWL,  Ft  Holabird 
1 USA  SE  Signal  Sch.  Ft  Gordon.  ATTN:  ATSO-EA 
1 USA  Chaplain  Ctr  & Sch.  Ft  Hamilton.  ATTN:  ATSC-TE  RD 
1 USATSCH.  Ft  Eustis,  ATTN:  Educ  Advisor 

1 USA  War  College,  Carlisle  Barracks,  ATTN:  Lib 

2 WRAIR,  Neuropsychiatry  Div 
1 D L I.  SD A.  Monterey 

1 USA  Concept  Anal  Agcy,  Bethesda.  ATTN:  MOCA  WGC 
1 USA  Concept  Anal  Agcy,  Bethesda,  ATTN:  MOCA-MR 
1 USA  Concept  Anal  Agcy,  Bethesda.  ATTN:  MOCA  JF 
1 USA  Artie  Test  Ctr.  APO  Seattle,  ATTN:  STEAC-MO-ASI 
1 USA  Artie  Test  Ctr.  APO  Seattle.  ATTN:  AMSTE  -PL  TS 
1 USA  Armament  Cmd,  Redstone  Arsenal.  ATTN:  ATSKTEM 
t USA  Armament  Cmd,  Rock  Island,  ATTN:  AMSAR-TDC 
1 FAANAFEC.  Atlantic  City.  ATTN:  Library 
1 FAA  NAFEC,  Atlantic  City.  ATTN:  Hum  Engr  Br 

1 FAA  Aeronautical  Ctr.  Oklahoma  City.  ATTN:  AAC-44D 

2 USA  Fid  Arty  Sch.  Ft  Sill.  ATTN:  Library 
1 USA  Armor  Sch,  Ft  Knox.  ATTN:  Library 

1 USA  Armor  Sch,  Ft  Knox.  ATTN:  ATSB-DI-E 
1 USA  Armor  Sch.  Ft  Knox.  ATTN:  ATSB-DT  TP 
1 USA  Armor  Sch.  Ft  Knox.  ATTN:  ATSB-CD  AD 


2 HQUSACDEC,  Ft  Ord,  ATTN:  Library 

1 HQUSACDEC.  Ft  Ord.  ATTN:  ATEC-EX-E-Hum  Factors 

2 USAEEC,  Ft  Benjamin  Harrison,  ATTN:  Library 

1 USAPACDC.  Ft  Benjamin  Harrison,  ATTN:  ATCP-HR 
t USA  Comm -Elect  Sch,  Ft  Monmouth,  ATTN:  ATSN-EA 
1 USAEC.  Ft  Monmouth.  ATTN:  AMSEL-CT-HDP 
1 USAEC.  Ft  Monmouth,  ATTN:  AMSEL-PA-P 
1 USAEC.  Ft  Monmouth.  ATTN:  AMSEL-SI-CB 
1 USAEC.  Ft  Monmouth.  ATTN:  C.  Facl  Dev  Br 
1 USA  Materials  Sys  Anal  Agcy.  Aberdeen.  ATTN:  AMXSY-P 
1 Edge  wood  Arsenal,  Aberdeen,  ATTN:  SAREA-BL-H 

1 USA  Ord  Ctr  & Sch,  Aberdeen.  ATTN:  ATSL-TEM-C 

2 USA  Hum  Engr  Lab,  Aberdeen,  ATTN:  Library/Dir 

1 USA  Combat  Arms  Tng  Bd,  Ft  Benning,  ATTN:  Ad  Supervisor 
1 USA  Infantry  Hum  Rsch  Unit,  Ft  Benning,  ATTN:  Chief 
1 USA  Infantry  Bd.  Ft  Benning,  ATTN:  STEBC-TE-T 
1 USASMA,  Ft  Bliss.  ATTN:  ATSS-LRC 
1 USA  Air  Def  Sch,  Ft  Bliss.  ATTN:  ATSA--CTD-  ME 
1 USA  Air  Def  Sch,  Ft  Bliss,  ATTN:  Tech  Lib 
1 USA  Air  Def  Bd,  Ft  Bliss,  ATTN:  FILES 
1 USA  Air  Def  Bd.  Ft  Bliss.  ATTN:  STEBD-PO 
I USA  Cmd  & General  Stf  College,  Ft  Leavenworth,  ATTN:  Lib 
1 USA  Cmd  & General  Stf  College,  Ft  Leavenworth,  ATTN:  ATSW-SE-L 
1 USA  Cmd  & General  Stf  College,  Ft  Leavenworth,  ATTN:  Ed  Advisor 
1 USA  Combined  Arms  Cmbt  Dev  Act.  Ft  Leavenworth,  ATTN:  DepCdr 
1 USA  Combined  Arms  Cmbt  Dev  Act,  Ft  Leavenworth.  ATTN:  CCS 
1 USA  Combined  Arms  Cmbt  Dev  Act.  Ft  Leavenworth,  ATTN:  ATCASA 
1 USA  Combined  Arms  Cmbt  Dev  Act,  Ft  Leavenworth,  ATTN:  ATCACO-E 
1 USA  Combined  Arms  Cmbt  Dev  Act,  Ft  Leavenworth,  ATTN:  ATCACC-CI 
1 USAECOM.  Night  Vision  Lab.  Ft  Belvoir,  ATTN;  AMSEL-NV-SD 

3 USA  Computer  Sys  Cmd,  Ft  Belvoir,  ATTN:  Tech  Library 
1 USAMERDC.  Ft  Belvoir.  ATTN:  STSFB-DQ 

1 USA  Eng  Sch,  Ft  Belvoir,  ATTN:  Library 
1 USA  Topographic  Lab,  Ft  Belvoir,  ATTN:  ETL  TD-S 
1 USA  Topographic  Lab.  Ft  Belvoir,  ATTN:  STINFO  Center 
1 USA  Topographic  Lab,  Ft  Belvoir.  ATTN:  ETL-GSL 
1 USA  Intelligence  Ctr  & Sch.  Ft  Huachuca,  ATTN:  CTD-MS 
1 USA  Intelligence  Ctr  & Sch,  Ft  Huachuca,  ATTN:  ATS— CTD— MS 
f USA  Intelligence  Ctr  & Sch,  Ft  Huachuca.  ATTN:  ATSI-TG 
1 USA  Intelligence  Ctr  & Sch.  Ft  Huachuca.  ATTN:  ATSI-TEX-GS 
\ USA  Intelligence  Ctr  & Sch,  Ft  Huachuca.  ATTN:  ATSI-CTS-OR 
1 USA  Intelligence  Ctr  & Sch.  Ft  Huachuca.  ATTN:  ATSi-CTD-DT 
1 USA  Intelligence  Ctr  & Sch.  Ft  Huachuca.  ATTN:  ATSI-CTD-CS 
1 USA  Intelligence  Ctr  & Sch,  Ft  Huachuca,  ATTN:  DAS/SRD 
1 USA  Intelligence  Ctr  & Sch,  Ft  Huachuca.  ATTN:  ATSI-TEM 
1 USA  Intelligence  Ctr  & Sch,  Ft  Huachuca,  ATTN:  Library 

1 CDR,  HQ  Ft  Huachuca.  ATTN:  Tech  Ref  Div 

2 CDR,  USA  Electronic  Prvg  Grd.  ATTN:  STEEP-MT-S 
1 CDR.  Project  MASSTER.  ATTN:  Tech  Info  Center 

1 Hq  MASSTER.  USATRADOC.  LNO 
1 Research  Institute.  HQ  MASSTER.  Ft  Hood 
1 USA  Recruiting  Cmd.  Ft  Sherdian.  ATTN:  USARCPM-P 
1 Senior  Army  Adv.,  USAFAGOD/T AC,  Elgin  AF  Aux  Fid  No.  9 
1 HQ  USARPAC,  DCSPER.  APO  SF  96558.  ATTN:  GPPE-SE 
1 Stimson  Lib,  Academy  of  Health  Sciences,  Ft  Sam  Houston 
1 M »rine  Corps  Inst.,  ATTN.  Dean -MCI 
1 HQUSMC.  Commandant,  ATTN:  Code  MTMT  51 

1 HQUSMC.  Commandant.  ATTN:  Code  MPI-20 

2 USCG  Academy,  New  London,  ATTN:  Admission 
2 USCG  Academy,  New  London,  ATTN:  Library 

1 USCG  Training  Ctr.  NY.  ATTN:  CO 
1 USCG  Training  Ctr,  NY.  ATTN:  Educ  Svc  Ofc 
1 USCG.  Psychol  Res  Br.  DC.  ATTN:  GP  1/82 
1 HQ  Mid-Range  Br.  MC  Det.  Quantico.  ATTN:  PftS  Div 


E-l 


ARI  TP  286 


1  US  Marin*  Corpt  Liaiaion  Ofc.  AMC,  Alexandria,  ATTN:  AMCGS-F 
1 USATRADOC.  Ft  Monro*.  ATTN:  ATRO-ED 
e USATRADOC.  Ft  Monro*.  ATTN:  ATPR-AO 
1 USATRAOOC,  Ft  Monro*.  ATTN:  ATTS-EA 

1 USA  Forces  Cmd.  Ft  McPherson.  ATTN:  Library 

2 USA  Aviation  Tut  Bd.  Ft  Rucker.  ATTN:  STEBG-PO 

1  USA  Agcy  for  Aviation  Saf*ty.  Ft  Ruckar,  ATTN:  Library 
1 USA  Agcy  for  Aviation  Safaty.  Ft  Ruck*r,  ATTN:  Educ  Advitor 
1 USA  Aviation  Sch,  Ft  Ruck*r,  ATTN:  PO  Drawer  O 

1 HQUSA  Aviation  Syt  Cmd.  St  Loud.  ATTN:  AMSAV-ZDR 

2 USA  Aviation  Syt  Teit  Act..  Edwardt  AFB.  ATTN:  SAVTE-T 
f USA  Air  D*f  Scfi,  Ft  Blit*.  ATTN:  ATSA  TEM 

1 USA  Air  Mobility  Rtcb  ft  Dm  Lab.  Moffttt  Fid.  ATTN:  SAVDL-AS 
1 USA  Aviation  Sch.  R*t  Tng  Mgt,  Ft  Ruckar.  ATTN:  ATST-T-  RTM 
t USA  Aviation  Sch.  CO.  Ft  Ruckar.  ATTN:  ATST-O-A 
f HO.  USAMC,  Alexandria.  ATTN:  AMXCD-TL 
t HQ.  USAMC.  Alexandria,  ATTN:  CDR 
1 US  Military  Acadamy.  Watt  Point  ATTN:  Sarialt  Unit 
1 US  Military  Acadamy.  Watt  Point,  ATTN:  Ofc  of  Milt  Ldrthp 
J US  Military  Acadamy.  Wait  Point  ATTN:  MAOR 
1 USA  Standardization  Gp.  UK.  FPO  NY.  ATTN:  MASE-GC 
I Ofc  of  Naval  Rteh.  Arlington,  ATTN:  Cod*  4S2 

3 Ofc  of  Naval  Rach,  Arlington,  ATTN:  Cod*  458 
1 Ofc  of  Naval  Rich,  Arlington,  ATTN:  Coda  450 
1 Ofc  of  Naval  Rach,  Arlington,  ATTN:  Cod*  441 

1 Naval  Aarotpc  Mad  Rat  Lab,  Paniacola,  ATTN:  Acout  Sch  Oiv 
f Naval  Aarotpc  Mad  Rat  Lab,  Pantacola,  ATTN:  Cod*  L51 
1 Navel  Aarotpc  Mad  Rat  Lab,  Pantacola,  ATTN:  Coda  L5 
I Chief  of  NavPart,  ATTN:  Pert  OR 
1 NAVAIRSTA.  Norfolk.  ATTN:  Safety  Ctr 
1 Nav  Oceanographic.  DC.  ATTN:  Coda  6251.  Charts  ft  Tech 
1 Cantor  of  Naval  Anal,  ATTN:  Doc  Ctr 
1 NavAirSytCom.  ATTN.  AIR-531X 
1 Nav  BuMed.  ATTN:  713 
1 NavHalicoptarSubSqua  2.  FPO  SF  86601 
1 AFHRL  (FT)  William  AFB 
1 AFHRL  <TT)  Lotvry  AFB 

1 AFHRL  (AS)  WPAFB,  OH 

2 AFHRL  (DOJZ)  Brooks  AFB 

1 AFHRL  (OOJN)  Lackland  AFB 
1 HQUSAF  (INYSO) 

1 HQUSAF  (DPXXA) 

1 AFVTG  (RD)  Randolph  AFB 

3 AMRL  (HE)  WPAFB.  OH 

2 AF  Intt  of  Taeh,  WPAFB.  OH.  ATTN:  ENE/SL 
1 ATC  (XPTD)  Randolph  AFB 

1 USAF  AaroMad  Lib,  Brook i AFB  (SUL-4),  ATTN:  DOC  SEC 
1 AFOSR  (NL).  Arlington 

1 AF  log  Cmd.  McClellan  AFB.  ATTN:  ALC/DPCRB 

1 Air  Fore*  Acadamy.  CO,  ATTN:  Dept  of  6*1  Sen 
6 ./Part  ft  Dm  Ctr,  San  Diago 

2 N ,v,  Me  f Nauroptychiatric  Rich  Unit  Sen  Diago 
' N«  ' .n-tronie  Lab,  San  Diago.  ATTN.  Rat  Lab 

’ h«v  TrngCan,  San  Diago,  ATTN:  Coda  8000-Lib 
1 Navf . ttGraSch,  Monterey,  ATTN.  Coda  SSA* 

I NavPottGraSch,  Montaray,  ATTN:  Coda  2124 
1 NavTrngEquipCtr,  Orlando,  ATTN:  Tech  Lib 
1 US  Dapt  of  Labor.  DC.  ATTN:  Manpower  Admin 
1 US  Dapt  of  Justice,  DC,  ATTN:  Drug  Enforce  Admin 
I Nat  Bur  of  Standards.  DC,  ATTN:  Computer  Info  Section 
1 Nat  Clearing  Horn*  tor  MH-Info,  Rockville 
1 Denver  Federal  Ctr.  Lakewood.  ATTN:  BLM 
12  Dafanaa  Documanutton  Canter 

4 Dir  Psych,  Army  Hq.  Ruaaatl  Ofet.  Canberra 

1 Scientific  Advtr,  MM  Bd.  Army  Hq.  RuastH  Ofet.  Canberra 
I Mil  and  Air  Attache,  Ausbien  Embassy 

1 Centra  da  Racharch*  Das  Factaurt.  Humain*  da  la  Oatonsa 
National*.  Brutaeb 

2 Canadian  Joint  Staff  Washington 

1 C/Arr  Staff.  Royal  Canadian  AF,  ATTN  Part  Std  Anal  ftr 

3 Chief.  Canadian  Daf  Rteh  Staff.  ATTN:  C/CRDSfW) 

4 British  Daf  Staff.  British  Embassy,  Washington 


1 Daf  ft  Civil  Intt  of  Enviro  Madicin*.  Canada 
1 AIR  CRESS,  Kensington,  ATTN:  Info  Syt  Br 
1 Militeerptykologitk  Tjenatta,  Copahagsn 
I Military  Attach*.  French  EmLttty.  ATTN:  Doc  Sac 
1 Madacin  Chat,  C.E.R.PJk.-Artena:,  Toulon/Naval  Franca 
1 Prin  Scientific  Off,  Appl  Hum  Engr  Rich  Oiv,  Ministry 
of  Defense.  New  Delhi 

1 Pars  Rteh  Ofc  Library,  AKA.  Israel  Defense  Forest 
1 Minuter  is  van  Defensis,  DOOP/KL  Afd  Sociaal 
Psychologitche  Zaken,  The  Hague,  Nether  lands 


* 


